<?xml version="1.0" encoding="ISO-8859-1"?><article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<front>
<journal-meta>
<journal-id>1414-4077</journal-id>
<journal-title><![CDATA[Avaliação: Revista da Avaliação da Educação Superior (Campinas)]]></journal-title>
<abbrev-journal-title><![CDATA[Avaliação (Campinas)]]></abbrev-journal-title>
<issn>1414-4077</issn>
<publisher>
<publisher-name><![CDATA[Publicação da Rede de Avaliação Institucional da Educação Superior (RAIES), da Universidade Estadual de Campinas (UNICAMP) e da Universidade de Sorocaba (UNISO)]]></publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id>S1414-40772021000200568</article-id>
<article-id pub-id-type="doi">10.1590/s1414-40772021000200012</article-id>
<title-group>
<article-title xml:lang="en"><![CDATA[A correct method to calculate the IDD value-added indicator in the Brazillian higher education quality assurance framework]]></article-title>
<article-title xml:lang="pt"><![CDATA[Um método correto para calcular o indicador de valor agregado IDD do Sistema Nacional de Avaliação da Educação Superior]]></article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author">
<name>
<surname><![CDATA[Haar]]></surname>
<given-names><![CDATA[Ewout ter]]></given-names>
</name>
<xref ref-type="aff" rid="Aff"/>
</contrib>
</contrib-group>
<aff id="Af1">
<institution><![CDATA[,University of São Paulo Institute of Physics ]]></institution>
<addr-line><![CDATA[São Paulo SP]]></addr-line>
<country>Brazil</country>
</aff>
<pub-date pub-type="pub">
<day>00</day>
<month>05</month>
<year>2021</year>
</pub-date>
<pub-date pub-type="epub">
<day>00</day>
<month>05</month>
<year>2021</year>
</pub-date>
<volume>26</volume>
<numero>2</numero>
<fpage>568</fpage>
<lpage>586</lpage>
<copyright-statement/>
<copyright-year/>
<self-uri xlink:href="http://educa.fcc.org.br/scielo.php?script=sci_arttext&amp;pid=S1414-40772021000200568&amp;lng=en&amp;nrm=iso"></self-uri><self-uri xlink:href="http://educa.fcc.org.br/scielo.php?script=sci_abstract&amp;pid=S1414-40772021000200568&amp;lng=en&amp;nrm=iso"></self-uri><self-uri xlink:href="http://educa.fcc.org.br/scielo.php?script=sci_pdf&amp;pid=S1414-40772021000200568&amp;lng=en&amp;nrm=iso"></self-uri><abstract abstract-type="short" xml:lang="en"><p><![CDATA[This paper investigates the value-added indicator used in the Brazilian higher education quality assurance framework, the so-called IDD indicator for undergraduate programmes (&#8220;Indicator of the difference between observed and expected outcomes''). The two main claims are that since 2014 this indicator is calculated incorrectly and that this mistake has relevance for public policy. INEP, the educational statistical agency responsible for educational quality indicators in Brazil, incorrectly uses multilevel modeling in their value-added analysis. The IDD indicator is calculated by estimating a varying intercept linear mixed model, but instead of identifying the intercepts with the value added of courses, INEP uses the mean of the student residuals. That this was indeed the error made is shown by reproducing exactly INEP&#8217;s published values using the incorrect method with the microdata for the 2019 assessment cycle. I then compare these values to the ones obtained with the same model and same data, but using the correct value-added measure. A comparison of reliability estimates for both methods shows that this measure of internal consistency is indeed higher for the correct method. As an example of policy relevance, I calculate the number of courses that would change from &#8220;satisfactory&#8221; to &#8220;unsatisfactory&#8221; and vice-versa, using the usual criteria established by INEP, if the correct method is applied.]]></p></abstract>
<abstract abstract-type="short" xml:lang="pt"><p><![CDATA[O trabalho investiga o indicador de &#8220;valor agregado&#8221; usado no sistema de avaliação do ensino superior Brasileiro, o chamado indicador IDD (Indicador de Diferença entre os Desempenhos Observado e Esperado) para cursos de graduação. As duas principais afirmações defendidas são que desde 2014 este indicador é calculado de forma equivocada pelo INEP (Instituto Nacional de Estudos e Pesquisas Educacionais) e que este equívoco teve consequências relevantes para as políticas públicas. O INEP usa modelagem multinível para calcular o IDD, usando um modelo linear misto com interceptos aleatórios. Em vez de identificar o IDD com os interceptos estimados, o INEP usa a média dos residuais no nível dos estudantes. É possível reproduzir exatamente os valores dos IDDs publicados pelo INEP a partir dos microdados do ciclo de 2019 usando o método equivocado. Uma comparação dos valores do IDD do INEP com aqueles obtidos usando o método correto mostra índices de confiabilidade maiores usando o método correto. Como exemplo de uma consequência relevante, determino o número de cursos que foram equivocadamente classificados com IDD &#8220;satisfatório&#8221; segundo os critérios usuais do INEP.]]></p></abstract>
<kwd-group>
<kwd lng="pt"><![CDATA[Ensino superior]]></kwd>
<kwd lng="pt"><![CDATA[Valor agregado]]></kwd>
<kwd lng="pt"><![CDATA[Indicadores de qualidade]]></kwd>
<kwd lng="pt"><![CDATA[IDD]]></kwd>
<kwd lng="en"><![CDATA[Higher education]]></kwd>
<kwd lng="en"><![CDATA[Value-added modeling]]></kwd>
<kwd lng="en"><![CDATA[Quality indicators]]></kwd>
<kwd lng="en"><![CDATA[IDD]]></kwd>
</kwd-group>
</article-meta>
</front><back>
<ref-list>
<ref id="B1">
<nlm-citation citation-type="journal">
<collab>AERA; APA; NCME</collab>
<article-title xml:lang=""><![CDATA[Standards for educational and psychological testing]]></article-title>
<source><![CDATA[American Educational Research Association]]></source>
<year>2014</year>
<publisher-loc><![CDATA[Washington, DC ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B2">
<nlm-citation citation-type="journal">
<collab>AERA</collab>
<article-title xml:lang=""><![CDATA[AERA Statement on Use of Value-Added Models (VAM) for the Evaluation of Educators and Educator Preparation Programs]]></article-title>
<source><![CDATA[Educational Researcher]]></source>
<year>2015</year>
<volume>44</volume>
<numero>8</numero>
<issue>8</issue>
<page-range>448-52</page-range><publisher-loc><![CDATA[California ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B3">
<nlm-citation citation-type="">
<collab>AMERICAN STATISTICAL ASSOCIATION</collab>
<source><![CDATA[ASA statement on using value-added models for educational assessment]]></source>
<year>2014</year>
<publisher-loc><![CDATA[Virginia ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B4">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[BATES]]></surname>
<given-names><![CDATA[D.]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Fitting Linear Mixed-Effects Models Using lme4]]></article-title>
<source><![CDATA[Journal of Statistical Software]]></source>
<year>2015</year>
<volume>67</volume>
<numero>1</numero>
<issue>1</issue>
<publisher-loc><![CDATA[Los Angeles ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B5">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[BITTENCOURT]]></surname>
<given-names><![CDATA[H. R.]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Uma análise da relação entre os conceitos Enade e IDD]]></article-title>
<source><![CDATA[Estudos em Avaliação Educacional]]></source>
<year>2008</year>
<volume>19</volume>
<numero>40</numero>
<issue>40</issue>
<page-range>247</page-range><publisher-loc><![CDATA[São Paulo ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B6">
<nlm-citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname><![CDATA[CASTELLANO]]></surname>
<given-names><![CDATA[K. E.]]></given-names>
</name>
<name>
<surname><![CDATA[HO]]></surname>
<given-names><![CDATA[A. D]]></given-names>
</name>
</person-group>
<source><![CDATA[A practitioner&#8217;s guide to growth models]]></source>
<year>2013</year>
<publisher-loc><![CDATA[Washington ]]></publisher-loc>
<publisher-name><![CDATA[CCSSO]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B7">
<nlm-citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname><![CDATA[FERNANDES]]></surname>
<given-names><![CDATA[R.]]></given-names>
</name>
</person-group>
<source><![CDATA[Avaliação de cursos na educação superior: a função e a mecânica do conceito preliminar de curso]]></source>
<year>2009</year>
<publisher-loc><![CDATA[Brasília ]]></publisher-loc>
<publisher-name><![CDATA[Inep]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B8">
<nlm-citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname><![CDATA[FERNANDES]]></surname>
<given-names><![CDATA[V]]></given-names>
</name>
</person-group>
<source><![CDATA[Quality of accounting undergraduate programs in Brazil: how to estimate value-added?]]></source>
<year>2020</year>
<publisher-loc><![CDATA[Uberlândia ]]></publisher-loc>
<publisher-name><![CDATA[Universidade Federal de Uberlândia]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B9">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[IKUTA]]></surname>
<given-names><![CDATA[C. Y. S]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Sobre o conceito preliminar de curso: concepção, aplicação e mudanças metodológicas]]></article-title>
<source><![CDATA[Estudos em Avaliação Educacional]]></source>
<year>2016</year>
<volume>27</volume>
<numero>66</numero>
<issue>66</issue>
<page-range>938</page-range><publisher-loc><![CDATA[São Paulo ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B10">
<nlm-citation citation-type="book">
<collab>INEP</collab>
<source><![CDATA[O que é o Sinaes]]></source>
<year></year>
<publisher-loc><![CDATA[Brasília ]]></publisher-loc>
<publisher-name><![CDATA[Inep]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B11">
<nlm-citation citation-type="book">
<collab>OECD</collab>
<source><![CDATA[Indicadores de qualidade da educação superior. Outros documentos]]></source>
<year>2020</year>
<month>a</month>
<publisher-loc><![CDATA[Brasília ]]></publisher-loc>
<publisher-name><![CDATA[Inep]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B12">
<nlm-citation citation-type="book">
<collab>INEP</collab>
<source><![CDATA[NOTA TÉCNICA No 34/2020/CGCQES/DAES]]></source>
<year>2020</year>
<month>b</month>
<publisher-loc><![CDATA[Brasília ]]></publisher-loc>
<publisher-name><![CDATA[Inep]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B13">
<nlm-citation citation-type="book">
<collab>INEP</collab>
<source><![CDATA[Microdados do indicador da diferença entre os desempenhos observado e esperado]]></source>
<year>2020</year>
<month>c</month>
<publisher-loc><![CDATA[Brasília ]]></publisher-loc>
<publisher-name><![CDATA[Inep]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B14">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[JOLLY]]></surname>
<given-names><![CDATA[E]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Pymer4: Connecting R and Python for Linear Mixed Modeling]]></article-title>
<source><![CDATA[Journal of Open Source Software]]></source>
<year>2018</year>
<volume>3</volume>
<numero>31</numero>
<issue>31</issue>
<page-range>862</page-range><publisher-loc><![CDATA[United States ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B15">
<nlm-citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname><![CDATA[KIM]]></surname>
<given-names><![CDATA[H.]]></given-names>
</name>
<name>
<surname><![CDATA[LALANCETTE]]></surname>
<given-names><![CDATA[D]]></given-names>
</name>
</person-group>
<source><![CDATA[Literature review on the value-added measurement in higher education]]></source>
<year>2013</year>
<publisher-loc><![CDATA[Paris ]]></publisher-loc>
<publisher-name><![CDATA[OECD Publishing]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B16">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[KORETZ]]></surname>
<given-names><![CDATA[D]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Measuring postsecondary achievement: lessons from large-scale assessments in the K-12 Sector]]></article-title>
<source><![CDATA[Higher Education Policy]]></source>
<year>2019</year>
<volume>32</volume>
<numero>4</numero>
<issue>4</issue>
<page-range>513-36</page-range><publisher-loc><![CDATA[London ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B17">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[LACERDA]]></surname>
<given-names><![CDATA[L. L. V. DE]]></given-names>
</name>
<name>
<surname><![CDATA[FERRI]]></surname>
<given-names><![CDATA[C.]]></given-names>
</name>
<name>
<surname><![CDATA[DUARTE]]></surname>
<given-names><![CDATA[B. K. DA C]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[SINAES: avaliação, accountability e desempenho]]></article-title>
<source><![CDATA[Avaliação]]></source>
<year>2016</year>
<volume>21</volume>
<numero>3</numero>
<issue>3</issue>
<publisher-loc><![CDATA[Campinas; Sorocaba ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B18">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[LACERDA]]></surname>
<given-names><![CDATA[L. L. V. DE]]></given-names>
</name>
<name>
<surname><![CDATA[FERRI]]></surname>
<given-names><![CDATA[C]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Conceito preliminar de curso: conceito único para uma realidade educacional múltipla]]></article-title>
<source><![CDATA[Estudos em Avaliação Educacional]]></source>
<year>2017</year>
<volume>28</volume>
<numero>69</numero>
<issue>69</issue>
<page-range>748-72</page-range><publisher-loc><![CDATA[São Paulo ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B19">
<nlm-citation citation-type="book">
<collab>OECD</collab>
<source><![CDATA[Assessment of higher education learning outcomes. Feasibility study report]]></source>
<year>2013</year>
<volume>3</volume>
<publisher-loc><![CDATA[Paris ]]></publisher-loc>
<publisher-name><![CDATA[OECD Publishing]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B20">
<nlm-citation citation-type="book">
<collab>OECD</collab>
<source><![CDATA[Rethinking quality assurance for higher education in Brazil]]></source>
<year>2018</year>
<publisher-loc><![CDATA[Paris ]]></publisher-loc>
<publisher-name><![CDATA[OECD Publishing]]></publisher-name>
</nlm-citation>
</ref>
<ref id="B21">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[POLIDORI]]></surname>
<given-names><![CDATA[M. M]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Políticas de avaliação da educação superior brasileira: Provão, SINAES, IDD, CPC, IGC e outros índices]]></article-title>
<source><![CDATA[Avaliação]]></source>
<year>2009</year>
<volume>14</volume>
<numero>2</numero>
<issue>2</issue>
<page-range>439-52</page-range><publisher-loc><![CDATA[Campinas; Sorocaba ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B22">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[PRIMI]]></surname>
<given-names><![CDATA[R.]]></given-names>
</name>
<name>
<surname><![CDATA[SILVA]]></surname>
<given-names><![CDATA[M. C. R. da]]></given-names>
</name>
<name>
<surname><![CDATA[BARTHOLOMEU]]></surname>
<given-names><![CDATA[D]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[A validade do Enade para avaliação de cursos superiores]]></article-title>
<source><![CDATA[Examen: política, gestão e avaliação da educação]]></source>
<year>2018</year>
<volume>2</volume>
<numero>2</numero>
<issue>2</issue>
<page-range>128-51</page-range><publisher-loc><![CDATA[Brasília, DF ]]></publisher-loc>
</nlm-citation>
</ref>
<ref id="B23">
<nlm-citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname><![CDATA[STEEDLE]]></surname>
<given-names><![CDATA[J. T]]></given-names>
</name>
</person-group>
<article-title xml:lang=""><![CDATA[Selecting value-added models for postsecondary institutional assessment]]></article-title>
<source><![CDATA[Assessment &amp; Evaluation in Higher Education]]></source>
<year>2012</year>
<volume>37</volume>
<numero>6</numero>
<issue>6</issue>
<page-range>637-52</page-range></nlm-citation>
</ref>
<ref id="B24">
<nlm-citation citation-type="">
<collab>TCU. Tribunal de Contas da União</collab>
<source><![CDATA[Relatório de auditoria (RA): RA 01047120170]]></source>
<year>2018</year>
<publisher-loc><![CDATA[Brasília ]]></publisher-loc>
</nlm-citation>
</ref>
</ref-list>
</back>
</article>
