GHA 1654-9716 1654-9880 Global Health Action Vol. 4, December 2011, pp. 1–11 sici xxx PUBMED Abbreviation Richard Wootton

*Professor R Wootten, Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway NO-9038 Tromsø Norway

Anton Vladzymyrskyy Maria Zolfo Laurent Bonnardot Norwegian Centre for Integrated Care and Telemedicine University Hospital of North Norway

Tromsø Norway

Department of Informatics and Telemedicine Donetsk R&D Institute of Traumatology and Orthopedics

Donetsk Ukraine

Department of Clinical Sciences Institute of Tropical Medicine

Antwerp Belgium

Department of Medical Ethics Paris Descartes University

Paris France

Fondation Médecins Sans Frontières

Paris France

Global Health Action ORIGINAL ARTICLE Experience with low-cost telemedicine in three different settings Telemedicine has been used for many years to support doctors in the developing world. Several networks provide services in different settings and in different ways. However, to draw conclusions about which telemedicine networks are successful requires a method of evaluating them. No general consensus or validated framework exists for this purpose. To define a basic method of performance measurement that can be used to improve and compare teleconsultation networks; to employ the proposed framework in an evaluation of three existing networks; to make recommendations about the future implementation and follow-up of such networks. Analysis based on the experience of three telemedicine networks (in operation for 7–10 years) that provide services to doctors in low-resource settings and which employ the same basic design. Although there are many possible indicators and metrics that might be relevant, five measures for each of the three user groups appear to be sufficient for the proposed framework. In addition, from the societal perspective, information about clinical- and cost-effectiveness is also required. The proposed performance measurement framework was applied to three mature telemedicine networks. Despite their differences in terms of activity, size and objectives, their performance in certain respects is very similar. For example, the time to first reply from an expert is about 24 hours for each network. Although all three networks had systems in place to collect data from the user perspective, none of them collected information about the coordinator's time required or about ease of system usage. They had only limited information about quality and cost. Measuring the performance of a telemedicine network is essential in understanding whether the network is working as intended and what effect it is having. Based on long-term field experience, the suggested framework is a practical tool that will permit organisations to assess the performance of their own networks and to improve them by comparison with others. All telemedicine systems should provide information about setup and running costs because cost-effectiveness is crucial for sustainability. telemedicine telehealth developing countries performance evaluation Network Countries Model Swinfen Charitable Trust Various Targeted Ukraine Mainly Ukraine Targeted Institute of Tropical Medicine Mainly Africa Targeted iPath Various Open Partners Healthcare, Boston, USA Cambodia Targeted Tripler Army Medical Center, Honolulu, USA Mainly Pacific region Targeted Telemedicine appears to be useful in the context of the developing world 2 and store-and-forward teleconsultation is one of the main applications used in this context 3. But which is the best model for teleconsultation? Which networks are performing well and which are not? To answer these questions, we need a framework for evaluating network performance. Frameworks exist in other domains (e.g. business, health improvement), but these are not really appropriate for our purposes and there does not appear to be a validated framework for evaluation and assessment of telemedicine systems. Thus, the objectives of the present study were: to define a basic method of performance measurement that can be used to improve and compare teleconsultation networks; to employ the proposed framework in an evaluation of three existing networks; and to make recommendations about the future implementation and follow-up of such networks. In accordance with the three objectives, the work was conducted in three stages, as follows. To develop a suitable framework for measuring network performance, we first defined the concept. Then, we defined the actors. Finally, we defined the measurement perspective. This allowed the framework itself to be defined. Measuring the performance of a system is one aspect of its overall evaluation. A telemedicine network does not exist in isolation. It is a component or sub-system within an organisation. The performance of the organisation as a whole can be represented by its capacity to fulfil four functions: to realise its mission, for example to achieve its goals; to acquire resources, such as money and prestige, and adapt them to its needs; to produce services of appropriate quality and volume; and to maintain and develop its internal values, that is its organisational culture. In practice, there will be a dynamic tension between the achievement of these four functions4. A telemedicine network can, therefore, be viewed as a tool for achieving the third of the four organisational functions. How then can the performance of such a system be measured? To evaluate any system requires: selecting the characteristics to be measured; choosing a suitable method to measure these characteristics; collecting the data; analysing the collected data; making decisions on the basis of the results; and implementing those decisions. Thus, performance measurement encompasses steps 2 3 4 . It may involve measuring how the system works (i.e. process measures) or what it achieves (i.e. outcome measures). Process measures assess the general operation of the system. Here, the focus is on the ability of the system to conduct its work. This may entail examining whether the necessary resources exist (e.g. funding, staff and equipment) to achieve the planned activities. A qualitative method is likely to be the most appropriate technique. Outcome measures assess the changes or benefits that result from the activities of the system. Typically, quantifiable measures are used to determine whether the system has achieved a certain level of performance. In the present context, the aim is to understand both whether the telemedicine network is working as intended and what effect it is having. Therefore, both process and outcome measures are required (even though outcome measures may be difficult to obtain in practice and need time before the effects can be measured). The choice of performance measures may be facilitated by considering a telemedicine network as a ‘black box’. A black box is a device or system that can be viewed solely in terms of its input, output and transfer characteristics without any knowledge of its internal workings. Almost anything might be referred to as a black box, e.g. a transistor, an algorithm or a hospital. In the context of telemedicine network performance, the network (black box) has an input (the queries that are made) and an output (the answers provided), see Fig. 1. So that the black box can perform its function, resources have to be supplied. In the present context, these resources (principally the experts) are controlled by one or more telemedicine coordinators.

A telemedicine network viewed as a black box.

For the purpose of the present article, we describe the characteristics of a particular form of telemedicine called teleconsultation. The service is provided via a network operated by volunteers, although the principles are also relevant in networks operated by staff who are paid to do the work. Thus, we define a teleconsultation as taking place when: a requester, such as a referring doctor, requests information from an expert. In practice, the requester makes the request to the telemedicine network, rather than to a specific expert; a telemedicine coordinator receives the request, evaluates it and decides which would be the appropriate expert to answer it. (In the open model, the coordinator may allocate the request to a group of experts.) The coordinator then notifies the chosen expert that a query is waiting to be dealt with. Each request may generate multiple queries, for example if the first specialist to whom the query is sent is not available, then the query will need to be sent to another. In practice, because the coordinator is likely to require both medical and IT skills, the coordinator is often supported by one or more assistants; and an expert, such as a medical specialist, receives a request for information and (hopefully) responds to the requester. Thus, the present article is concerned with three types of persons (requesters, coordinators and experts) and how to measure their performance, although not in a punitive way. In attempting to measure performance, it is helpful to consider whose perspective should be used. This is because the desirable performance from the point of view of the requester may be very different from that of the expert, or the coordinator. For example, the requester might hope for an immediate reply, whereas the specialist – who is likely to have many other demands on his or her time – may wish to deal with requests at leisure. User perspective. The first perspective is that of the network users, who all have the same desire: a reliable and easy to use system, for which the minimum of training is required. Thus, the requester would like: a rapid answer; a definitive response to the questions posed (‘definitive’ meaning comprehensive, appropriate and adapted to the local context); the possibility of discussing the case with the specialist, if the answer received is not definitive; and additional information, if relevant, such as references from the literature. The coordinator would like: a ‘clean’ referral, that is one without poor quality pictures or information contained in files of proprietary format. There may also be a requirement to receive carefully anonymised referral data if confidentiality is critical, as in humanitarian operations in war zones; access to a panel of experienced and available specialists, covering all sub-specialities; and a rapid response from the specialist, even if the response is only that a different specialist will be required. The expert would like: only appropriate referrals, that is requests that fall within his or her areas of expertise; sufficient information about the patient to permit an appropriate response to be provided; information about the local context to provide a reply that is well adapted to circumstances; and feedback from the referrer about the reply and/or subsequent follow-up data about the patient. Societal perspective. The other perspective that is relevant is that of society as a whole. Telemedicine is not simply a tool but a procedure allowing interactions between a network and the local health care systems. So, to measure the impact of the network, we must take into consideration the context, Fig. 2. Telemedicine may create a new way of accessing health care, for example obtaining a second opinion in remote areas or in a humanitarian setting, where the population often has no access to specialist physicians. This is a completely different matter from telemedicine that improves or optimises an existing health care system, for example telemedicine projects set up in industrialised countries. The distinction is important because it has direct consequences in terms of evaluation: in the latter case, a comparison between face-to-face and telemedicine consultation can be done, for example, whereas in the former case, this is not possible because there is no previously existing reference system.

Overview of a telemedicine system. Telemedicine networks are not isolated but located within larger health organisations that in turn are part of the general environment.

The societal aims will, therefore, be a telemedicine network that provides clinically useful information at a ‘reasonable’ cost. That is, the use of telemedicine will contribute to strengthening the local health care system. Patient's perspective. Finally, the patient's perspective should not be forgotten, as patient care is the ultimate goal of any telemedicine service. Even if the patient and requester both aspire to obtain the best possible health care management, their perception of what is best can differ. The doctor can take advantage of expert advice (e.g. from an educational point of view), whereas the patient would find no benefit at all. Patient satisfaction must, therefore, be assessed independently, and this requires a specific qualitative approach. Despite these potential differences between the views of requester and patient, we consider in the present context that the patient's perspective and the requester's perspective are tightly linked. There is a wide range of indicators and metrics that might be relevant to the measurement of performance. Indicators ‘indicate’ impact but they do not attempt to quantify that impact, whereas metrics are ‘numerical indicators’ that allow the impact to be quantified. A combination of indicators and metrics is, therefore, needed. Because of the complexity of the environment in which telemedicine networks operate, both quantitative and qualitative methods are needed, and a multi-disciplinary approach is required for analysing results. In particular, a qualitative approach allows a deeper understanding of system use and, therefore, a better measurement of its true impact and so an improved assessment of quality. As far as the performance of the network as a whole is concerned, important metrics include: the average speed of response to requests, that is, the average delay before a requester receives a definitive reply; and the cost of providing the service, for example, the coordinator's time, the infrastructure required. As far as performance of the network for the individual users is concerned, important metrics include: the ‘value’ of the answer provided, for example, was it relevant (in accordance with the most up-to-date medicine adjusted by the expert's experience)? Did it help the patient? How long did it take? Did significant knowledge transfer occur between requester and expert (but also for the coordinators)? Although a large number of possible indicators and metrics might be employed to answer these questions, in our experience five measures in each of the three user groups will suffice, that is, after discussion, it appears that these five measures capture the majority of the information required. There are basically five characteristics of the system that a requester needs to know about, in advance of submitting a new referral: the rate at which queries are being submitted to the network (which tells the referrer how busy the network is and how well it is working); the proportion of failed queries (which tells the referrer what the chance of getting a reply is); the time to reply (which tells the referrer how long they will have to wait); the quality of the reply (which tells the referrer how good the replies are). The definition of ‘quality’ will include how useful the information is to the requester, how well adapted it is to the requester's local situation and also whether it follows the most recent guidelines as well as the educational value of the answer (knowledge transfer); and how easy the system is to use. The things that a coordinator needs to know are: How many questions will need to be handled? (Strictly speaking, what is the rate of arrival of requests for me to deal with?) How much time and effort will this entail? What resources are available to me? For example, how many experts are there, and what subject areas do they cover? What will I do if I do not have an appropriate expert for a particular query? Note that this leads to the question whether a more effective form of global telemedicine would be a network of networks. In other words, should there be a single, high-level clearing house for receiving telemedicine queries, from which queries could be directed to the participating networks as appropriate? Will I be able to obtain feedback from the users, for example, from the experts? Will I be able to obtain feedback on patient outcomes? How easy is the system to use? Are there likely to be IT problems and if so, how will these be solved? The things that an expert needs to know are: How many queries will I receive? That is, what is the rate of work requests that I can expect? How much time and effort will be required to answer them? (Note that responding to a single query about a clinical case may be much less demanding than answering a generic query that may require providing the referrer with an informal summary of the literature.) How much time do I have to respond to the query? Will the queries be relevant to my area of expertise? Will I receive feedback about the patient outcomes? How easy is the system to use? For example, can I notify periods when I am not available to answer requests because of absence due to holidays or conferences? Some of the above items are aspects of quality, including the technical quality of attached images. Finally, from the societal perspective, the aspects of system performance that are important are: Is the network clinically effective? That is, does it improve patient outcomes? Is the network cost-effective? That is, is it an appropriate use of scarce resources? Are the network and its participants properly integrated into the relevant health care organisations? To illustrate the use of the framework for measuring network performance, we applied it to the networks of our own experience. The authors are responsible for the operation of three telemedicine networks in different settings (low- and middle-income countries). The three networks all provide teleconsultation services, that is, they have a clinical purpose and have been in operation for periods of 7–10 years, see Table 2. 1. Swinfen Charitable Trust network This service is provided by a UK charity and started operations in 1999. It provides advice to doctors in developing countries using web-based store-and-forward messaging. In 2009, there were approximately 300 referrers from 45 developing countries. The advice is provided by a panel of about 400 consultants who donate their expertise to the charity. In the first 10 years of operation, the network handled approximately 2,000 telemedicine cases. 2. Ukrainian Teleconsultation network This service started in 2000. This network is based in an expert trauma centre in Donetsk and provides treatment advice to seven community hospitals in the Donetsk region and four regional centres in the Ukraine using web-based store-and-forward and low-cost videoconferencing (Skype) methods. In the first 10 years, more than 700 teleconsultations were conducted in trauma and orthopaedics. 3. Institute of Tropical Medicine HIV/AIDS Telemedicine network This service is based at the Institute of Tropical Medicine in Antwerp and started in 2003. It provides second opinion advice about the use of antiretroviral therapy and AIDS care delivery using email and web-based store-and-forward methods. Expert advice from a network of 20 specialists in HIV/AIDS is offered to health care workers in resource-limited settings. In the first 7 years, the telemedicine service provided over 1,000 teleconsultations to health workers in more than 40 countries, most located in sub-Saharan Africa. Each of the three networks has been evaluated at various times during its operation: Swinfen charitable trust network. Type of evaluation conducted: Regular audits of system activity have been performed. Other forms of evaluation have included surveys of user satisfaction, analysis of submitted image quality and follow-up of patients. Main results. Audits of system activity show that the time required to provide a specialist's response has remained at 2 days or less over 10 years of operations 5 6 7 ; surveys of user satisfaction show that the referring doctors who responded made positive comments about the service and half said that it improved their management of the case 8; analysis of image quality showed that although average image size increased over a 4-year period, the quality of the submitted images did not 9; follow-up of patients showed that telemedicine had assisted with the diagnosis in all cases and the clinical outcome for the patient was considered to be good in 15 of the 44 cases10. Ukrainian teleconsultation network. Type of evaluation conducted: A retrospective cohort study (n=210) was carried out in 2007–2009. The aim was to define the quality of the teleconsultations and their influence on clinical outcomes. Patients were divided into three groups (telemedicine and two control groups) 11. Main results. The majority of teleconsultations (92%) had high relevance. The experts’ recommendations had a real influence on the clinical strategy in 81% of the cases. There were more good clinical outcomes in the telemedicine patients studied than in a group of control patients (66% vs. 31%). An evaluation showed that telemedicine was associated with a sixfold increase in the odds of good clinical outcomes for trauma patients. Institute of tropical medicine HIV/AIDS telemedicine network. Type of evaluation conducted: A web-based survey was sent to network users in 2006 and in 2008 to assess their perception of the system and its value in establishing the diagnosis, changing the management of care, providing reassurance to the referring doctor and any educational benefit 12. Main results. The referring clinicians reported that the service was beneficial in several ways: Establishing the diagnosis (42%), clinician's education (27%), literature provision (14%) and for reassurance about decisions made prior to the consultation (12%). The service was judged to have been useful in influencing the management of the patients in 100% of cases. Most users (88%) reported that the service had significantly influenced the way they managed their cases and also other similar cases subsequently 13. The performance framework described above was applied to the three networks, using data available to the coordinators. The results are summarised in Table 3. Performance measure SCT Ukraine ITM Comment Value Comment Value Comment Value Statistics for the year 2009 Statistics for the year 2007–2009 Statistics from April 2003–March 2009 Requester's perspective 1. Rate of query arrival (new cases) 19.4/month 5.8/month 15.5/month 2. Proportion of failed queries 0 0 0 3. Time to first reply from an expert Median 17 hours 30 hours 24 hours 4. Quality of replies Some evidence from a research project (10). Referring clinicians reported that in 100% of cases followed up (n=22), the advice was useful Not measured routinely Some evidence from a research project (11). The quality of replies was measured according to a specially developed scale 20. Ninety-two per cent of replies (n=132) were judged to be highly relevant Not measured routinely Some evidence from a research project (12) User satisfaction (38 respondents) measured on a scale 1–10. Utility of advice was rated 8–9 by 70% of the respondents Not measured routinely 5. Ease of system usage Not measured Not measured Not measured Coordinator's perspective 1. Rate of query arrival New queries, plus those requiring additional responses 37.7/month New queries 5.8/month New queries. 15.5/month 2. Time required Not measured 15–30 min Not measured 3. Resources available 406 experts available Five experts available 16 experts available 4. Feedback from experts/feedback on patient outcomes Not measured Some evidence from a research project (11). Feedback was available on the relevance of the teleconsultation for 63% of the cases (n=132) Feedback was available on patient outcomes for 33% of the patients (n=132) Not measured routinely Not measured 5. Ease of system usage Not measured Not measured Not measured Expert's perspective 1. Rate of requests received (for those experts who received queries) 2.8/year per expert 5.4/year per expert 9.9/year per expert 2. Time required to answer Median time from allocation to response 11.0 hours 30 min Not measured 3. Relevance to own expertise Not measured Not measured Not measured 4. Feedback on patient outcomes Some evidence from a research project (10) Not measured routinely Some evidence from a research project (11) Not measured routinely Not measured 5. Ease of system usage Not measured Not measured Not measured Societal perspective 1. Clinical-effectiveness Some evidence from a research project (10). Referring clinicians reported that the telemedicine advice was useful in 21 of 22 cases followed up, and the outcome for the patient was good in 15 cases Not measured routinely Some evidence from a research project (11). Referring clinicians reported that telemedicine produced satisfactory outcomes in 22% of cases (n=132) and good or excellent outcomes in a further 76% Not measured routinely Some evidence from a research project (13). Referring clinicians reported that telemedicine was useful in influencing the management of their patients in 100% of cases (n=37) Not measured routinely 2. Cost-effectiveness Not measured Not measured Not measured 3. Integration into the health care system, for example, involvement of local people Not measured Not measured Not measured The following analysis is based on the experience of running three second opinion telemedicine networks for periods of 7–10 years and on an evaluation of the performance of the three networks using the proposed framework, that is, the aim is to offer evidence-based advice about practical implementation. Although the three networks are different in terms of their activity, their size and their objectives, their performance in certain respects is very similar. For example, the time to first reply from an expert is about 24 hours for each network, see Table 3. However, in illustrating the use of the framework to assess the performance of the three telemedicine networks, it became clear that only some of the information required was in fact available. All networks had systems in place to collect certain data from the user perspective. None of them collected information about the coordinator's time required or about ease of system usage (for any of the three user types). They had only limited information about cost and quality. We recommend that networks collect such information routinely. Information about the coordinators’ time was not collected by any of the networks. Nor was information collected about usability. Measuring ease of system usage requires data collection directly from the field to see how the system works in practice from the perspective of its users. This requires a qualitative approach, based on well-defined tools such as observation (participative or not), interviews (in depth, semi structured or guided discussion) or scale/questionnaires 14. Considering the major impact of the environment on system use, only such a pragmatic approach can provide insights into potential causal mechanisms of dysfunction and, therefore, generate hypotheses to improve the system. Mixed methods have been used successfully in outcomes research in other disciplines 15 and, therefore, seem appropriate in telemedicine as well. We recommend that data on system operation and usability be collected routinely. The issue of cost is less important in networks run by volunteers. However, estimation of costs is likely to be required in future operations where staff are reimbursed. Clearly, rigorous cost-effectiveness studies are best performed by an expert, for example, a health economist16 17, but some estimation of cost seems essential in the context of a telemedicine operation that relies on technology. A recent review of telemedicine in developing countries3 showed that only a few articles provided such data although in the context of scarce resources, the appropriate allocation of resources is crucial (i.e. the consequences of wastage that will have little effect on health care in industrialised countries may have a profound impact in low-resource settings). Cost details also provide global information on the system, which can be useful for comparisons with other systems. We recommend that cost data be collected routinely. The quality of the replies was not systematically assessed by any of the three example networks – nor has it been reported more generally in other telemedicine networks – despite being fundamental to the main value of a telemedicine system. Content analysis of replies needs to be performed by independent experts, and ideally within the framework of a multi-disciplinary evaluation (medical, sociopsychological and anthropological) to take fully into account the environment and its impact on patient management. This kind of data collection has to be planned beforehand, which in turn emphasises the need for a standardised framework when setting up a new project. User satisfaction, including patient feedback is another aspect of the quality of the replies. Simple and reliable tools are available, although they may need to be adapted to suit the specific system being studied. The use of a numerical scale by Zolfo et al. 13. – that was based on the visual-analogue scale used for assessing pain – represents a useful example of the adaptation of validated tools for assessing user satisfaction. We recommend that quality of service data be collected routinely. As well as an absence of data about the user perspective, the three networks collected very little information about network performance from the societal perspective. In a study of telehealth adoption in Quebec hospitals, Gagnon et al. 18 found that the local context was a crucial factor in long-term success. Understanding the local context requires a field approach and feedback from local users. Close collaboration with local national health care workers is necessary to make sure that any proposed telemedicine system will address real needs and meet local priorities. Only limited data were available on clinical effectiveness and none at all on cost-effectiveness. Measuring health outcomes (e.g. mortality and morbidity, diagnostic accuracy) is difficult in resource-limited settings, especially in situations where health is influenced by a range of factors that are difficult to control for. Measuring cost-effectiveness is, therefore, likely to be extremely difficult, although some limited information about costs might be obtainable. We recommend that data be collected routinely to enable measurement of clinical- and cost-effectiveness. Based on our experience, the targeted model of telemedicine network operation appears to be successful and sustainable. The three networks we have used to illustrate the proposed framework are different in terms of their activity, their size and their objectives, but their performance in certain respects is very similar. On the other hand, we cannot conclude that the targeted model is superior to the open model without further evidence such as performance measurements from a wider range of networks. A telemedicine network can be regarded as an example of a ‘diffuse technology’, rather like a digital picture-archiving system (PACS). As was pointed out during the evaluation of early PACS networks, there are ways of combining individual studies to arrive at strong evidence about the impact of a diffuse technology 19. For example, information from individual case studies can be combined, so that even if they represent relatively weak evidence individually, the aggregated value is strong. Thus, two or three separate case studies based on different data sources that show similar effects will help to confirm the value of a diffuse technology. Our hypothesis is that stronger evidence about the value of telemedicine networks in the developing world can be obtained from studies of multiple networks than from a study of any network alone. We have proposed a performance measurement framework for assessing telemedicine networks that provide teleconsultation services to doctors in resource-constrained settings, although the principles are more general and could be applied to other telemedicine networks as well. The suggested performance measures for the three types of users show certain common, cross-cutting themes. All users are concerned with: Access (how busy is the network? what will my workload be?). Quality (how good are the responses? can feedback be obtained about patient outcomes?). Usability (how easy is the system to use? what will I do if there are technical problems?). Similar themes can be seen in the societal perspective. The framework is based on a core set of indicators and metrics. We have illustrated its use to assess the performance of three mature telemedicine networks. The results help us to understand whether these telemedicine networks are working as intended and the effect that they are having. Telemedicine is a well-known minefield. It is a recipe for disaster to begin a pilot project without having a satisfactory method for evaluating network performance. We recommend that everyone uses a common framework. This will facilitate comparisons between networks. More important, it allows those responsible to make longitudinal comparisons within their own networks, for example, to assess the effects of improvements. We also recommend that all networks collect basic information about their costs. Measurement of performance is a fundamental aspect of network evaluation, yet there is no existing framework in the literature that is relevant in the present context. Conceptualising a telemedicine network as a ‘black box’ allows the identification of important indicators and metrics that are common to the different perspectives. While examining these different perspectives (user and societal), we propose a framework for performance evaluation based on five main items from each perspective. This framework was applied to three different telemedicine networks, and certain common performance characteristics emerged. It also became clear that some important data were not measured at all, possibly because acquiring this information would require a specific field approach. For a comprehensive evaluation that takes into account the environment, mixed methods are required, which combine both quantitative and qualitative methods. Cost-effectiveness analysis must be performed by an expert, but all telemedicine systems should at least provide information about setup and running costs. The present article describes the performance of three different telemedicine networks, operating in low- and middle-income countries. The performance framework is based on our experience. No telemedicine network should be started without having evaluation built in. Our suggested framework will permit an organisation to understand both whether the telemedicine network is working as intended and what effect it is having. The authors have no conflicts of interest, such as financial or personal relationships, that could bias the work reported here. Bonnardot L Rainis R. Store-and-forward telemedicine for doctors working in remote areas J Telemed Telecare 2009 15 1 6 Wootton R. Telemedicine support for the developing world J Telemed Telecare 2008 14 109 14 Wootton R Bonnardot L. In what circumstances is telemedicine appropriate in the developing world? JRSM Short 2010 1 37 Lafortune L Farand L Mondou I Sicotte C Battista R. Assessing the performance of health technology assessment organizations: a framework Int J Technol Assess Health Care 2008 24 76 86 Wootton R Youngberry K Swinfen R Swinfen P. Referral patterns in a global store-and-forward telemedicine system J Telemed Telecare 2005 11 100 3 Swinfen P Swinfen R Youngberry K Wootton R. A review of the first year's experience with an automatic message-routing system for low-cost telemedicine J Telemed Telecare 2003 9 63 5 Graham LE Zimmerman M Vassallo DJ Patterson V Swinfen P Swinfen R et al. Telemedicine – the way ahead for medicine in the developing world Trop Doct 2003 33 36 38 Wootton R Youngberry K Swinfen P Swinfen R. Prospective case review of a global e-health system for doctors in developing countries J Telemed Telecare 2004 10 94 96 Jakowenko J Wootton R. An analysis of the images attached to referral messages in an email-based telemedicine system for developing countries J Telemed Telecare 2006 12 49 53 Wootton R Menzies J Ferguson P. Follow-up data for patients managed by store and forward telemedicine in developing countries J Telemed Telecare 2009 15 83 88 Vladzymyrskyy AV , Klymovytskyy VG , Kalinovsky KD , Dorokhova ET Clinical efficiency of telemedicine in trauma and orthopaedics. In: Jordanova M, Lievens F, eds. Med-e-Tel Exhibition and Conference Guide. Luxembourg: Med-e-Tel Forum; 16–18 April 2008, pp. 68–69. Zolfo M Bateganya MH Adetifa IM Colebunders R Lynen L. A telemedicine service for HIV/AIDS physicians working in developing countries J Telemed Telecare 2011 17 65 70 Zolfo M Renggli V Koole O Kiyan C de Waard I Lynen L. Telemedicine for HIV/AIDS care in resource-limited settings and users’ satisfaction Med-e-Tel Global Telemedicine and eHealth updates: knowledge resources Jordanova M Lievens F Med-e-Tel Grimbergen Belgium 2008 15 18 Brooke J . SUS: a quick and dirty usability scale. In: Jordan PW , Thomas B , Weerdmeester BA , McClelland ML ,. Usability evaluation in industry. London : Taylor and Francis ; 1996, pp. 189–194. Curry LA Nembhard IM Bradley EH. Qualitative and mixed methods provide unique contributions to outcome research Circulation 2009 119 1442 52 Johnston K Kennedy C Murdoch I Taylor P Cook C. The cost-effectiveness of technology transfer using telemedicine Health Policy Plan 2004 19 302 9 Martinez A Villarroel V Puig-Junoy J Seoane J del Pozo F. An economic analysis of the EHAS telemedicine system in Alto Amazonas J Telemed Telecare 2007 13 7 14 Gagnon MP Lamothe L Fortin JP Cloutier A Godin G Gagné JP Telehealth adoption in hospitals: an organisational perspective J Health Organ Manag 2005 19 32 56 Keen J Bryan S Muris N Weatherburn G Buxton M. Evaluation of diffuse technologies: the case of digital imaging networks Health Policy 1995 34 153 66 Vladzymyrskyy AV. Telemedicine consultations in daily clinical practice: systems, organisation, efficiency Handbook of research on distributed medical informatics and E-health Lazakidou AA Siassiakos KM Information Science Reference Hershey PA 2008 260 72