News
Reflecting on an Experiment in Technology Evaluation at MIT

With the MIT Comprehensive Initiative for Technology Evaluation (CITE) entering its fifth year, I sat down with Ford International Professor of Urban Planning and CITE’s Director Bishwapriya Sanyal, to gather his thoughts on what the program has accomplished over its tenure, and what lessons one could learn about how to address developmental challenges drawing on CITE’s research

McDevitt: How was the Comprehensive Initiative on Technology Evaluation created at MIT? 

Sanyal: It essentially started with a course I taught with Amy Smith at MIT’s D-Lab This was a very popular course on Technology and Development which Amy had started. I came to learn about this course by reading a full page article in the New York times about Amy who had just received a MacArthur Fellowship. It was inspiring to read how Amy had been working to 
enhance the quality of life of very poor families by deploying small scale, cheap, but very effective technologies. Since I came from an academic background in which I was educated to understand much larger macro level questions about persistent poverty, deepening income inequalities and underdevelopment, I sensed in Amy’s action based research an alternative effort at the grassroots level to directly address poverty issues and improve the living condition of the poor. In other words, Amy’s micro level approach to poverty alleviation was in sharp contrast to the generally pessimistic assessment of all sorts of macro “structural barriers” which I was quite familiar and which in some ways had disempowered me, until then, to think pragmatically about how to address the needs of the poor. I was cognizant of the limits of micro level strategies of the kind Amy was advocating, but I saw in such efforts a modest aspiration that the quality of life of the poor can be made better, incrementally, and more importantly that such efforts can make us –the professionals—learn about what is important for the poor, how they made decisions, and what kind of technologies are they willing to pay for. It is true that poverty is caused by many interconnected factors and that is why it is so hard to break out of the poverty trap; but the process of coping with poverty can be eased a bit if the poor can get potable water, breathe a less polluted air by, say, using better designed cookstoves, and similar other small scale interventions which Amy was experimenting with. 

As I taught with Amy for a few years, MIT’s D-lab started expanding its portfolio of research steadily, and similar efforts to design products for the poor started springing up not just in the US, but also around the globe. As a result, numerous products, many badly designed, flooded the market in the name of helping the poor. That is why D-Lab decided to create a research unit to be focused on evaluation of products. One key assumption underlying this effort at evaluation was: that there was no shortage of gadgets and various technological inventions to help the poor. In fact, the market was flooded with products. What was lacking was rigorous evaluation of such products so that the poor could make more informed choices about whether to buy, what to buy, and which product met their needs the best. Just as D-lab was trying to build up its evaluation unit, MIT received two large scale research grants from USAID, one to be led by Amy on ways of fostering product innovation, and the other to be led by me on the creation of a new methodology for evaluation of products designed for the poor. USAID had just started a new Global Development Lab to utilize technological know-how for poverty alleviation, and MIT, along with six other top ranking research universities were selected through a worldwide and intensely competitive process. Each of the seven universities of the Global Developmental lab was to conduct research on a specific issue related to the best way technological know-how could benefit the poor. CITE, which I headed, was to responsible for generating a robust and rigorous methodology for product evaluation. 

McDevitt: Were there any organizations before CITE came into the field that were doing this kind of work? That were doing product evaluations in global development?

Sanyal: There had been some research on the evaluation of “appropriate technologies” which had emerged in the 1970s and 1980s but such evaluations had been somewhat ad hoc, and were not motivated by our goal–namely to create a methodology of the kind that Consumer Reports uses so effectively in informing decision making. Until CITE was created most researchers and practitioners were interested primarily in innovating products; very rarely were such products evaluated even though everyone knew that most products either did not reach the poor, or were used only for a few times before the poor discarded such products and went back to their old ways. There was a widespread acknowledgement that very few innovations were disseminated widely but no one had studied this issue in a serious way. Most of the literature in the field focused on innovations for mass consumption in Western nations; how to spread cheap and effective technologies for the vast majority of the poor in developing countries who lived with one, or at most two dollars a day had not received much attention. 

At CITE we created a cross-disciplinary team of scholars from within MIT drawing on research expertise in developmental planning, mechanical engineering, transportation and logistics, sociotechnical systems, and business administration. The reason we created such a team with diverse expertise is because we wanted to examine three different aspects, namely, suitability, scalability, and sustainability of products.

McDevitt: What do those three different aspects look like in practice? 

First, by evaluating suitability we assessed the extent to which a product actually delivers on its promise. For example, if you tell me that you have designed a solar lantern that will provide illumination for say 15 hours at a stretch, without recharging, we would check the validity of that claim. Does the solar lantern really work for 15 hours at a stretch? And how powerful is the illumination? How long does it take to recharge the batteries? And so on. 

There are other ways to assess suitability of any product. Let’s say a claim is made by the manufacturer that a solar lantern is water resistant, or that it is built in such a way that it will not break easily. At CITE, we would actually test the lanterns, drop water on them, and do all sorts of tests, somewhat similar to how Consumer Reports would do assessments in the United States. It was our hope that by the end of the project there would be something equivalent to a consumer report for poor consumers in developing countries. 

Next, CITE analyzed what we called “scalability”. The key underlying question here is to probe why even well designed products did not reach the poorest of the poor. In searching for an answer to this question, CITE tried to understand various factors which influence how product markets work in practice: Where do the inputs come from? How is the product delivered? Which stores sell such products? Do the stores accept installment payments? What does it cost the stores to store such products, and so on. One obvious question is: Can the product be repaired easily? These sort of questions provide a composite picture of why some products reach the poor while others do not. 

Our third criteria was “sustainability”. Here our focus was on why only some products are used over a long period of time. This has to do with not only product design–meaning, is the design user friendly—but also, cultural appropriateness, nature of ownership-individual or collective—and many other variables which ultimately affect whether a product is likely to be used for a long time. 

McDevitt: What were your goals for the program? 

The original goal was to tap the multidisciplinary expertise to create synthesized comparative evaluation scores for a family of products. The three criterion—namely, suitability, scalability and sustainability—were to be assessed separately and then these assessments would be combined in a composite score which then could be used by the consumer. We did not assume that it will only be the poor themselves. We were aware that international aid agencies, national and local governments, and NGOs often have to procure products very quickly to address devastations created by natural calamities as well as human generated conflicts. Such procurement decisions could be made more rationally if comparative evaluation of products was available. 

Another implicit goal of USAID’s Higher Education Solutions Network was to create a new intellectual linkage between USAID and top ranking universities. As a key player in the international development scene, USAID wanted to work closely with innovators as well as evaluators and help create a new cadre of young scholars and practitioners who would carry forward the new effort to put technological knowhow for the benefit of the poor. By providing generous grants to top level universities, USAID wanted to tap the best young minds as well as faculty to work on developmental issues in the particular way which was important to USAID. Hence there was much stress in the project for large scale student involvement.

McDevitt: What are some of the most interesting things you learned over the course of the program? 

In our study of solar lanterns, which is the first product family we selected to evaluate in Uganda, we learnt that the most important criteria for the poor as they select a solar lantern is not which products last longer or provide good illumination. What is of the most importance is which lantern charges their cell phone the fastest and for the longest time! This was an eye opener for me and the team as we realized that being connected to family, friends, and doctors may be more important for the poor than having good illumination.

True, Solar lanterns provide illumination, but the poor have lived without such illumination for a long time and in the process have developed a set of household practices which do not rely much on illumination at night. As one respondent noted, daily household work can be organized efficiently if one knows that there will be no illumination at night. But if there is no phone connection, and someone in the family gets very sick, how would one seek professional help? Such observations made me realize that there are many aspects of poor people's’ daily habits and practices that we know very little about. We have a long way to go before we can say confidently how poor households make resource allocation decisions.  

McDevitt: The final thing I want to touch on involves looking at the field of development as it moves forward, especially the ever increasing role of big data.

Sanyal: Certain kinds of problems require large scale data to better understand patterns of behavior. When one is analyzing, say transportation problems, one must know the pattern of when and where different groups of people travel to, what time do they leave home or work, which mode of transport they select, how many and for how long do they drive—understanding these sort of pattern questions requires large scale, city and regional level observations which in turn requires big data. With computerized ways of collecting and processing data, such as with sensors, it can be feasible and useful to have large data sets. 

That said, I don’t think that all developmental challenges—particularly, how to eradicate poverty—have been so difficult simply because of lack of data. Most developmental policies fail at the implementation level because as I mentioned earlier we really do not understand the preferences of the poor. Neither do we fully understand the factors that affect organizational performance. Can we generate better insights about such complex processes simply through large data sets? The point is to know which data is crucial, easy to collect, and politically safe to put it in the public domain. One other thing I learned in managing the CITE project is that there is no end to how much data can be collected. It is very expensive to collect good data and it is equally expensive to process that data rigorously. True, computers have lowered the cost but they have not reduced the complexity of the multifaceted nature of mass poverty. If our goal is to better understand that complex process so as to devise better policies, we need to understand 
social reality at least at three levels: at the levels of individuals, organizations within which such individuals operate, and the larger political-economy issues, such as political and economic systems. Big data can provide a glance at these interconnected issues, but by itself it cannot 
solve problems such as mass poverty. 

There are also downsides to large scale data collection, as I mentioned earlier. Yes, sensors have really helped, and we are using sensors to assess wheelchair performance, or use of solar pumps by salt miners in Gujarat, India. But if you asked me how expensive is the data gathering operation, and who will have the resources for large scale data collection, I would tell you that invariably large scale and somewhat centralized organizations, not NGOs, are likely to succeed, if at all. I doubt many governments can support large scale data collection. Should one be spending the meagre resources on data collection, or should it be spent on expanding the water delivery systems? One could of course argue that extension of water delivery can benefit from large scale data collection, but then the obvious question is: data about what? We do need good data on impact of public policies, which Randomized Control Trials (RCT) do provide, but even conducting a rigorous RCT can be quite expensive and time consuming. Most importantly, at the end of such a test you may realize that you do not understand the preferences of the poor. What then? Larger data sets, or different kind of 
data? 

The last thing I will say on large scale data collection is that who controls that data is bound to be a key political issue. It is my hunch that big organizations, big governments, and powerful militaries will have a competitive advantage on collecting large scale data and they will control it. 

McDevitt: How do you think CITE’s approach will change over time as emerging markets shift, grow, and mature? 

Sanyal: I think our three analytical lenses ---suitability, scalability and sustainability--will continue to be useful in any assessment method. The assessment of suitability is the easiest because one can test in a lab whether a product is made well. But here too we learned a lesson on the fact that how a product performs in a lab is not the same as how it may perform in a real life situation, even if one can simulate real life circumstances within a lab. And to understand what constitutes real life experiences one needs to know the key variables which influence the way poor families live, purchase, and use products. Whether a product meets consumers’ needs will remain an important factor to assess. In an emerging market economy where some consumers have more purchasing power than others, the emphasis in understanding may shift from the poor to the newly emerging middle class. The connection between retail, wholesale, delivery of the goods, some of these factors which constitute the supply chain may become more clearly articulated and hence will be easier to assess. One issue which is bound to remain a point of debate is regarding scales of operation. There is a current bias against “economy of scale” arguments in poverty alleviation efforts. As a result small scale operations are sometimes preferred over large scale operations, except in data collection. But as I learned while studying solar lanterns in Uganda, when China introduced a mass produced solar lantern at one third the cost of small scale, locally produced solar lanterns, the poor opted for the Chinese lanterns, particularly if such lanterns charged their cell phones better.