Measuring the completeness of consecutive interpretation of military subjects

Glushko Elena Valentinovna ORCID: 0000-0002-8412-5729 PhD in Philology Associate Professor; Department of Linguistics and Translation Studies; Moscow State Institute of International Relations (University) of Ministry of Foreign Affairs of the Russian Federation, Odintsovo Branch Novo-Sportivnaya str., 3, Odintsovo, Moscow region, 143007, Russia e.glushko@odin.mgimo.ru

Orlova Valentina Vladimirovna ORCID: 0009-0001-4327-0102 Teacher; Foreign Language Chair; Prince Alexander Nevsky Military University of the Ministry of Defense of the Russian Federation 14 Bolshaya Sadovaya str., Moscow, 125047, Russia v.orlova.nir@gmail.com

The relevance of this study is due to the fact that in modern translation studies, the problem of assessing the quality of interpretation is one of the key ones, especially in those industries in which the fate of people and entire states depends on the accuracy of formulations. One of these areas is military translation, which is turning from a tool for semantic and cultural transformation of the language code into a key factor in bilingual communication, on which global decisions at the highest state level depend. Due to the specifics of the work of military translators, they are required to ensure the fullest possible understanding of the original message of the recipients of the translation and to prevent distortion of incoming information, and in this process, achieving functional equivalence may not be enough ^{[1, p. 125]}. The main difficulty of translating military texts (both oral and written) remains achieving the maximum identity of the translation text to the original message ^{[2; 3]}. Accordingly, in order to assess the qualifications of a military interpreter and for the qualitative training of such specialists in consecutive military interpretation, it is necessary to develop an objective method for assessing the completeness of the translation.

To achieve the above-mentioned research goal, it is necessary to solve the following tasks:

- to analyze the existing methods of assessing the quality of consecutive interpretation;

- Identify key methods suitable for assessing the completeness of sequential military interpretation and determine which of them can provide objective data.;

- identify metrics based on which the completeness of consecutive interpretation can be calculated;

- perform test calculations.

The theoretical basis of the research consists of works in the field of assessing the accuracy and completeness of translation by leading Russian and foreign researchers. So, V. N. Komissarov, L. K. Latyshev, V. I. Provotorov and R. K. Minyar-Beloruchev propose to evaluate the accuracy of translation due to the presence or absence of errors in the language design or errors in the transmission of the content as a whole ^[4]. E. V. Alikina evaluates the quality of translation by analyzing "negative material", which includes omission of facts, distortions of logic and facts, additions, repetitions ^[5]. The team of authors of the monograph, edited by V. A. Mityagina, introduces criteria for evaluating the completeness of a translation, such as complete/incomplete transmission of basic information, clarifying information, and secondary information ^[4]. E. A. Alekseeva describes criteria for evaluating translation practice within the framework of the interpretive theory of translation, which, in the context of completeness of translation, include compliance with the ideas of the text and compliance with logical relations between ideas ^[6]. J. Lee introduces the concept of "accuracy" in the developed translation evaluation system, which takes into account omissions, additions and unjustified or erroneous changes to the original text ^[7]. E. A. Knyazheva, E. A. Pirko attempt an objective assessment of the translation using system analysis and hierarchy analysis method ^[8]. C. Chen also evaluates the completeness of the translation based on the theory of informativeness of R. K. Minyar-Beloruchev ^[9]. Based on the same theory, the completeness of the transmission of precision information in oral sequential translation is investigated by T. A. Volkova and A. S. Kovrova ^[10]. O. V. Albukova provides an analysis of the existing systems for quantifying the quality of translation ^[11].

To solve the tasks set in this work, the following research methods were used: the structural method for determining the boundaries of semantic units in the source text, the comparison method for comparing the semantic units used in the source text and in the translation text, the registration method of quality assessment for calculating the number of different categories of semantic units used by the translator during the translation, the empirical method to test the hypothesis on real examples of translation, a criterion analysis for categorizing semantic units used by the translator and deriving metrics, statistical analysis for examining arrays of semantic units and calculating their share in the total volume of meanings of the source text.

The research material was a statement by the former Minister of Defense of the Russian Federation, Army General Sergei Shoigu, at a conference call with the leadership of the Armed Forces on April 02, 2024 at the Situation Center of the Russian Ministry of Defense, published on the website of the Ministry of Defense of the Russian Federation.

The practical significance of the study lies in an attempt to optimize an objective assessment of the quality of consistent military interpretation. This method may be of interest in the fields of developing computer automation models for evaluating translation, evaluating the effectiveness of various methods of recording information. In addition, special attention should be paid to the assessment of the qualifications of interpreters, as well as the training of consecutive military and political interpreters, whose field of activity requires special accuracy of their translation due to the global impact of their product on the life and activities of both the state as a whole and an individual citizen. The methodology proposed by the authors can be used in reference studies to develop a quality standard for sequential military interpretation. All of the above points to the interdisciplinary nature of the current research.

Obviously, the most important characteristic of any translation is its adequacy, which means the most optimal translation quality within a specific communicative and substantive situation. At the same time, we must not forget that according to experts who practice military translation and teach this discipline in higher educational institutions, one of the distinctive features of military translation is detailed accurate translation and almost zero use of figurative and emotional expressive means, which certainly requires a higher level of equivalence in translation compared to other types of special translation.^[12]. In this regard, the perception of the translation text as an analogue of the original in the source language is more relevant for military translation than for any other type of special translation. Accordingly, consistent oral military translation significantly increases the importance of the maximum achievable semantic and pragmatic equivalence.

At the same time, a characteristic feature of sequential interpretation is the "absence of the material nature of information" ^{[13, p. 242]}. This means that the message is delivered exclusively orally and once, it is linear and time-limited. In this regard, the translation should be carried out as quickly and completely as possible without the possibility of contacting an electronic translator, dictionary, consultant specialist or additional sources of information. In comparison with other forms and types of translation (for example, with written or simultaneous translation), reproduction of the original message in full form with sequential translation is significantly difficult. One of the most important requirements for consistent military interpretation is the extremely accurate and clear transmission of the original message. This is especially true in the context of active hostilities, urgent negotiations, or in extreme situations. So, E. R. Svetlichnaya emphasizes that it is the accuracy of reproduction of the information received that directly affects the adequacy of the translation, to increase the level of its equivalence to the original statement and to achieve its functional purpose ^{[14, p. 147]}. The accuracy of reproduction of the received information can also be understood as the accuracy of reproduction of the meaning of the text, which directly depends on the accurate transmission of units of meaning. Consequently, omissions, additions, inaccuracies and distortion of meaning can lead not only to disruption of communication within the framework of the designated discourse, but also to more global contradictions between the parties, and consequently to a possible increase in the number of casualties among the civilian population, personnel of the armed forces and military equipment during the armed conflict.

The exact reproduction of units of meaning in translation studies is commonly referred to as the completeness of translation. We are guided by the following definition: completeness of translation is a characteristic of reproducing the meaning of the original message in the target language with similar semantic content, taking into account the presence or absence of omissions, distortions and additions in the translation text. The key element of the completeness of the translation is the semantic content of the message, not its lexical and syntactic structure, that is, its form.

Many prominent Russian and foreign scientists have been engaged in the analysis of the problems of assessing the quality of consecutive interpretation. However, not all researchers pay attention to the criterion of completeness of consecutive interpretation as a quality factor. At the same time, it is the assessment of the completeness of the transfer of meaning, in our opinion, that makes it possible to determine whether the translator has achieved semantic equivalence.

The problem of assessing the completeness of a translation (that is, the accuracy of information transmission in consecutive oral translation) is dealt with by such well-known domestic and foreign linguists as V. N. Komissarov, L. K. Latyshev, R. K. Minyar-Beloruchev, E. R. Svetlichnaya, E. V. Alikina, E. A. Alekseeva, authors of a collective monograph edited by V. A. Mityagina, teaching staff at the Marie Aps Institute of Translation at the University of Louvain (Belgium), J. Lee ^{[4, p. 304; 7, pp. 165-184; 14; 15, pp. 58-75]}. However, the key disadvantages of these works seem to us, firstly, the limited elaboration of criteria for assessing the completeness of translation, and, secondly, the fact that it is proposed to evaluate the quality of translation based on the expert opinion of a teacher or a group of teachers and translators, that is, fully dependent on the human factor. It turns out that the assessment of translation quality is highly subjective.

Noteworthy is the attempt to develop a method for objectively assessing the quality of translation by researchers such as E. A. Knyazheva and E. A. Pirko, who use system analysis and hierarchy analysis, which is one of the most productive methods of decision-making in general. The method involves mathematical processing of the results of an expert survey and determining the degree of consistency and reliability of the data obtained, thereby making the assessment of translation quality more objective ^[8]. However, firstly, the authors of the development do not use criteria directly related to the completeness of the translation, and secondly, the successful application of this technique requires the development of a mathematical model, which complicates this process when diagnosing an interpreter in a learning situation or during his self-diagnosis. In addition, the development of this methodology in itself proves the problem of inconsistency among experts evaluating the quality of translation, which means a high degree of subjectivity in this process, which can be significantly reduced through the use of mathematical models for evaluating the quality of translation, which is what the above-mentioned E. A. Knyazheva and E. A. Pirko strive to achieve.

It is known that objective methods of quality assessment, among others, include the method of statistical analysis and the registration method (GOST 15467-79, 1979). Accordingly, in order to increase the level of objectivity in assessing the quality of oral consistent military translation, it is necessary to calculate the amount of semantic information in the original message and in the completed translation. Next, obviously, it is necessary to calculate the ratio of the obtained values to each other. This process is proposed to be carried out by calculating the volume of the content of both messages.

Separately, I would like to focus on existing approaches to quantifying the quality of translation. In her review of existing approaches to the problem of assessing the quality of translation, O. V. Albukova cites several quantitative models for assessing the quality of translation, created for practical use by a number of organizations engaged in translation activities ^[11]. Such organizations include, in particular, the Canadian State Translation Bureau, SAE, the American Association of Translators, and others. Of course, taking into account the use of quantitative assessment of translation quality, these approaches are much more objective. However, according to the authors of this paper, the above-mentioned approaches to assessing the accuracy and, consequently, the quality of oral sequential translation seem to be insufficiently correct from the point of view of entering such a quantity as the number of words in the text. Due to the fact that the goal of any translation is, first of all, to achieve functional equivalence, it is not the specific words used in both the original and translated texts that play a much more significant role, but the meanings embedded in them. Thus, the number of semantic elements, which, in turn, can be conveyed by various semantic structures (individual words, phrases, or sentences), is proposed as the basic value against which certain inaccuracies in sequential translation are calculated.

Another method that deserves special attention is the calculation of the informative density coefficient proposed by E. V. Shelestyuk ^[16] based on the calculation of the informative coefficient by R. K. Minyar-Beloruchev ^{[15, p. 73]} and used by T. A. Volkova and A. S. Kovrova to evaluate the accuracy of sequential translation of precision information ^[10]. The authors propose to find the ratio of the amount of precision information to the number of syntagmas in the source text. Of the above, this method seems to be the most promising and correct due to the fact that the content of the text is a set of certain semantic units. It is based on the theory of text informativeness developed by R. K. Minyar-Beloruchev, which assumes the possibility of splitting the text into information quanta ("the smallest semantic segments of the text") and their further calculation to assess the quality of translation, which the author calls "calculating the informativeness of the text" ^[15].

In our study, we also propose to calculate the units of meaning, which in general linguistics are commonly called propositions. As V. N. Solovyova notes, the central element of a proposition is considered to be a predicate, which is mainly a verb ^[17]. In turn, A. S. Kravets presents the formula of the proposition in the form of the following statement: "Something/Someone (M) has the property of P" ^{[18, p. 73]}. At the same time, despite the fact that this formula may have a different lexical and syntactic structure, the content may be equivalent. In other words, it doesn't matter what lexical and semantic structure can be used to express, for example, the idea that the tank was moving slowly: These can be options such as "the speed of the tank was low", "the tank was moving at a low speed", "the speed of the tank was insignificant", etc. With any of them, the meaning, that is, the semantic content, is conveyed in full. Accordingly, it is quite acceptable to put a conditional sign of equality between the completeness of the conveyed meaning in each of the above propositions, that is, to consider them pragmatically equivalent to each other.

Thus, based on the above analysis, it can be concluded that it is possible to quantify the amount of meaning of any message, and accordingly, it can be processed using statistical methods. Therefore, such a characteristic of interpretation as its completeness can also be quantified.

This indicator may seem identical to such an indicator as the coefficient of informativeness, derived by R. K. Minyar-Beloruchev ^{[15, p. 73]}. However, it should be noted that the author calculates it only on the basis of the key information of the message, excluding additional, clarifying, repeated and null, using the category "communicative value of information". The authors of this study believe that, given the specifics of military translation, the content of a military text translation will have a high degree of completeness (accuracy) only when translated as closely as possible to the semantic content of the original message, that is, taking into account all the above types of information contained in the original text.

To calculate the completeness of an oral translation, we have developed an algorithm that allows us to identify not only the completeness of the translation, but also the most common mistakes made by the translator during the translation process. In other words, this algorithm makes it possible to identify not only the proportion of information accurately conveyed by the translator, but also the percentage of deviations, distortions and unjustified additions to the translation text, which can be of great importance both in assessing the qualifications of an interpreter and in his training. This algorithm includes the following steps::

1) dividing the original text into propositions and counting their number (T);

2) dividing the translation text into propositions (if necessary, during its preliminary transcription) and counting them (P);

3) counting generalizations (G), additions (A), inaccuracies (M), omissions (O) and cases of distortion of meaning, including contradictions (F);

4) calculation of the completeness of the translation of f using the formula:

5) calculation of the proportion of deviations from the meaning of d when translating using the formula:

6) calculation of the proportion of unjustified additions not included in the original message, a according to the formula:

7) calculation of the proportion of meaning distortions in the translation of e using the formula:

8) calculation of the proportion of omissions of meaning when translating about according to one of the formulas:

9) drawing up an opinion on the adequacy of the translation.

Thus, the above formulas are a calculation of the proportion of the number of meanings conveyed by the translator to the total number of meanings of the source text. Expressed as a percentage, the above metrics, in our opinion, give the clearest possible idea of the quality of translation and the degree of reliability and expertise of the translator in such a field as military interpretation.

As an example, here is the official translation of an excerpt from a speech by former Russian Defense Minister General of the Army Sergei Shoigu at a conference call with the leadership of the Armed Forces on April 02, 2024 at the Situation Center of the Russian Ministry of Defense (The Ministry of Defense of the Russian Federation, 2024).

The original message:

After the failure of the counteroffensive, the enemy is trying to gain a foothold on individual lines and positions, but has not achieved its goals in any of the directions. Groups of Russian troops continue to push the Ukrainian formations westward.

Official translation:

Having failed the counter-offensive, the enemy attempted to gain a foothold on certain lines and positions, but has not achieved any of its objectives in any direction. [...] Russian troops continue to push the enemy out of the occupied lines and reduce its combat capabilities.

Semantic units (propositions) used in the above text:

1) having failed the counter-offensive,

2) he enemy attempted to gain a foothold,

3) gain a foothold on lines,

4) gain a foothold on certain lines,

5) gain a foothold on positions,

6) has not achieved its objectives,

7) has not achieved any of its objectives,

8) has not achieved its objectives in any directions,

9) Russian troops push the enemy,

10) Russian troops continue to push the enemy,

11) to push the enemy out of the occupied lines,

12) continue to reduce its capabilities,

13) continue to reduce its combat capabilities.

Thus, in the original text above, it is possible to register the quantitative component of meaning in the amount of 13 units of meaning/propositions (T).

In transcribed form, this statement may look like this:

(1) Having failed the counter-offensive, (2) the enemy attempted to gain a foothold (4) on certain (3) lines and (5) positions, but (6) has not achieved (7) any of its objectives (8) in any direction. [...] (9) Russian troops (10) continue to push the enemy (11) out of the occupied lines and (12) reduce its (13) combat capabilities.

During a practical military interpretation class, one of the 2nd-year Master's students translated as follows:

After their counteroffensive, the enemy gained a foothold in some positions, but failed to achieve their goals. Russian troops have significantly pushed back enemy formations and increased their combat capabilities.

For an objective assessment of the quality of the translation performed, it is necessary to record the presence or absence of certain propositions from the original message in the translation text, as well as to note the use of generalization techniques, to record distortion of meaning, inaccuracies and unreasonable additions in the translation text. For example, as follows:

(1 F) Having failed the counter-offensive, (2 G) the enemy attempted to gain a foothold (4 M) on certain (3 O-) lines and (5 P+) positions, but (6 P+) has not achieved (7 O-) any of its objectives (8 O-) in any direction. [...] (9 P+) Russian troops (10 O-) continue to push the enemy (11 O-) out of the occupied lines and (12 F) reduce its (13 P+) combat capabilities. (A) Significantly.

Thus, we get that from the source text: precisely translated (P+) – 4 propositions, translated with generalization (G) - 1 proposition, omitted (O-) – 5 propositions, translated, but with inaccuracy (M) – 1 proposition, distorted meaning (F) in 2 propositions., added (A) – 1 proposition.

Applying the above formulas, we obtain a demonstration that the completeness of the translation is 38.5% of the original text of this fragment. The share of deviations from the meaning of the original fragment will be 46.2%, the share of distortions of meaning will be 15.4%, and the share of additions will be 7.7%. 38.4% of the original fragment data is omitted.

Separately, I would like to emphasize the need to distinguish between translational transformations used in translation and distortions of the source data. We recommend proceeding from the following principle: if additions, omissions, generalizations and other transformational techniques improve the overall adequacy of the translation text and achieve its communicative purpose, then such transformations, when using the proposed methodology for quantifying the quality of translation, should be perceived as actions that increase the completeness of the translation. At the same time, if they reduce the adequacy of the translation and distance the translator from achieving his communicative goal, then it is necessary to record distortions of the meaning of the original text and deviations from it.

In particular, it may be difficult to distinguish between the generalization of G and the unjustified omission of O information. In this case, it is necessary to refer to the definition of generalization, which is understood as a translation technique in which a similar appropriate generalization of the target language is used for the semantic unit of the source language. Thus, in a general sense, it is necessary that this technique contributes to the understanding of the meaning originally embedded in the text. In this case, the resulting translation can be called adequate, and the semantic unit can be added to the fully conveyed meanings.

Speaking about the other metrics, it is worth noting that distortion of the meaning of F implies the transmission of the opposite of the original meaning during translation or a meaning that significantly misleads the recipient; inaccurate transmission of the meaning of M implies an insignificant deviation from the meaning of the original message; unreasonable addition of A implies not supplementing the understanding of the source text by using additional lexical units (translation technique of lexical additions), and the addition of new data to the translation text that was not mentioned in the original text.

Thus, in the above example, we arrive at a fairly objective assessment of the completeness and correctness of professional translation: the recipient will receive less than 40% of the information that accurately or generically corresponds to the information in the source text, about 60% of the data will be received in a distorted form, 15% of which are erroneous and mislead the listener, as well as about 8% of the added information. by the translator of the information himself, which can also mislead the recipient. It is important to note that such a detailed examination of an ongoing translation can provide a translator (as well as a future translator during professional training) with a clearer understanding of the trajectory of his professional development.

Obviously, in order to assess the qualifications of a military interpreter and the quality of translation in general, it is necessary to analyze a much larger text. Nevertheless, the purpose of this work is to show a possible principle for quantifying the quality of consistent military interpretation, which can be extrapolated to the entire subsequent text.

At the same time, it should be noted that in order to assess the quality of an interpreter's translation and to assess his qualifications, such an analysis can be performed during the translation process if the expert evaluating the translation has the text of the original message and the propositions are pre-numbered in it. In the course of translation, the expert uses the above symbols to record the perceived accuracy of the translator's presentation of certain "ideas" (meanings, propositions) relative to the selected propositions in the source text, after which the proportion of accurately conveyed information, the proportion of omissions, the proportion of deviations, the proportion of distortions and the proportion of unreasonable additions are calculated by simple calculations.

To test this methodology, we analyzed more than 90 full-fledged translations of 3rd and 4th year undergraduate students and 1st and 2nd year graduate students studying at the Faculty of Linguistics and Intercultural Communication of MGIMO-Odintsovo. The research was based on socio-political texts with a duration of 1:58 and 2:58 minutes.

Based on the results obtained, it can be concluded that the proposed methodology for assessing the completeness of military interpretation gives a more objective idea of the accuracy of the translator's reporting of all the meanings used in the message than an expert opinion without relying on calculations. Despite the fact that an expert is involved in this technique, his participation is limited to the formal division of the text into semantic units (propositions) and making a decision on how accurately each specific semantic unit is conveyed in the translation.

In addition, taking into account the percentage of results obtained, it becomes easier to standardize the quality of translation services in the military sphere by setting limits on the amount of inaccurately transmitted information. Moreover, due to the fact that this analysis of the completeness of translation can be carried out not only for sequential translation, but also for written and oral simultaneous translations, therefore, quality standards can be developed for these types of translation.

Thus, we come to the following conclusions:

1) Using the above methodology for calculating the completeness of consecutive interpretation, the authors attempt to universalize and optimize an objective assessment of the quality of interpretation using the example of translating oral texts on military subjects due to its particular relevance in the current period of transformation of international relations in the world.

2) The theoretical and practical developments of domestic and foreign researchers in the field of assessing the quality of consecutive military interpretation and the completeness of translation, in particular, are often based on subjective expert assessment or on complex models for calculating the numerical component of translation quality, which makes it difficult to establish an objective assessment of both the quality of translation and the translator's qualifications.

3) Based on a comparison of the propositions in the original text and the translation text, it is possible to identify the quantitative component of the conveyed meanings in oral sequential military translation (and therefore other types of oral sequential translation), which can significantly increase the level of objectivity in assessing the quality of translation and the level of training and qualifications of interpreters, as well as help identify their most frequent errors in making a transfer.

4) For a comprehensive assessment of the completeness of the translation, the authors propose to calculate such indicators as the completeness of the translation, the proportion of deviations from the original meaning, the proportion of unjustified additions, the proportion of distortion of meaning, the proportion of omissions of meaning.

5) Using the example of a fragment of a military-political text, a demonstration was made of the application of the proposed methodology for assessing the completeness of an oral sequential translation and the intended conclusion with detailed data on each of the proposed metrics.

At this stage, the research is experimental and highly debatable. The method requires further testing and refinement. As prospects for further research of the stated issues, it is necessary to clarify the criteria for categorizing the meanings (propositions) used in translation, as well as to test the methodology in a number of experiments as an evaluation factor for the translator's qualifications, which can later be automated through the development of computer models for evaluating the quality of translation.