As in Chance v. Board of Examiners, 458 F.2d 1167 (2 Cir. 1972), and Bridgeport Guardians, Inc. v. Members of the Bridgeport Civil Service Comm'n, 482 F.2d 1333 (2 Cir. 1973), we are confronted with a claim that a city has unintentionally discriminated with respect to the employment of minority group members. Here, as in Chance, the city is New York; the alleged discrimination is with respect to persons allowed to qualify for the entering grade in the Fire Department. The action was brought under the Civil Rights Act, 42 U.S.C. 1983, and its jurisdictional counterpart, 28 U.S.C. 1343(3).^[1]

Plaintiffs, five minority individuals who had applied for employment with the Fire Department and two organizations representing minority firefighters, brought this suit as a class action in the District Court for the Southern District of New York. Their complaint alleged that the procedures used to select New York City firemen discriminated against blacks and Hispanics in violation of the equal protection clause of the Fourteenth Amendment. The defendants were the Civil Service Commission of the City of New York, the City's Department of Personnel, the chairman of the Commission and director of the Department, two members of the Commission, and then Fire Commissioner Lowery, hereafter referred to as the municipal defendants. Attacking on a broad front, plaintiffs first sought injunctive relief to prevent the Fire Department from making any more appointments based on an eligibility list reflecting performance on Exam 0159, a written civil service examination given on September 18, 1971. In addition, they sought to block further use of various other screening measures, including the Department's minimum height requirement, a requirement that every fireman have a high school or high school equivalency diploma, and the bar, arising from the combined effect of 487a-3.0(b) of chapter 19 of the Administrative Code of New York City and III-4.3.2(b) of the Rules and Regulations of its Civil Service Commission, against any applicant who had been convicted of a felony or of petty larceny.^[2]

At a hearing on plaintiffs' application for a preliminary injunction, Judge Weinfeld took evidence on the alleged discriminatory impact of the written examination and on the question whether the test was sufficiently related to a fireman's job to survive constitutional attack. At the conclusion of the seven-day hearing, he suggested that the parties agree to treat the hearing as a final trial on the merits of the case under F.R.Civ.P. 65(a)(2). Both parties agreed, but the plaintiffs requested that several points, including the automatic disqualification issues, be left open. Judge Weinfeld subsequently ruled that any further evidence on these points would have to be submitted before the court's decision was handed down, and that the decision would be final as to all issues in the case except for the challenge to the Fire Department's promotional examination, which would be left open for later consideration.

On June 12, in a comprehensive opinion, the district judge, 360 F.Supp. 1265, ruled that the written examination had a discriminatory impact and that it was not sufficiently job-related to justify its use. He enjoined further reliance on the challenged eligibility list, without prejudice to the parties' applying for interim relief which would permit appointments from the list on a quota basis until a new examination could be given and a new eligibility list established. Because the injunctive relief would benefit all persons similarly situated, Judge Weinfeld declined plaintiffs' request that he designate a class. As to the automatic disqualification issues, he ruled that it was unnecessary to consider those questions, stating:

This disposition makes it unnecessary to consider the other grounds urged by plaintiffs in support of their claim, particularly since little evidence was adduced with respect thereto upon the hearing. The submissions as to these matters were included in post trial briefs or affidavits and in some instances raise issues of fact, the resolution of which would require reopening of the trial.

He added there was serious doubt whether plaintiffs had standing to raise either the criminal conviction bar or the high school diploma requirement, since it appeared from the complaint that none of the named plaintiffs was subject to exclusion for those reasons.

Two months later, after having allowed intervention by non-minority candidates who had qualified under Exam 0159 but had not yet been appointed, the district court issued an order granting interim relief. The order instructed that in making future appointments from the challenged eligibility list, defendants would be required to hire one minority applicant for every three nonminority applicants hired. In an accompanying memorandum, the court further directed the municipal defendants 'to exert every good faith effort to accelerate the establishment of a new list.'^[3]

The defendants, intervenors, and plaintiffs all appealed from various portions of the district court's decision and order. We denied an application by the defendants and intervenors for a stay but brought the appeal here on an expedited basis. As is usual in cases of this sort, we have had a number of amicus briefs filed on behalf of various individuals and organizations. However useful amicus briefs may be on an issue of first impression in this circuit, see Chance, supra, 458 F.2d at 1169 & n.5, they only add to our burdens when the controlling principles have been established and the parties are so capably represented as here.

Under Chance and Bridgeport Guardians our analysis must be three-pronged. Was Judge Weinfeld 'clearly erroneous' in finding that Exam 0159 had had a 'racially disproportionate impact'?^[4] If not, did he err in concluding that the City had not made the requisite showing that Exam 0159 was sufficiently job-related; that is, did the City fail to prove that the disproportionate impact was simply the result of a proper test demonstrating lesser ability of black and Hispanic candidates to perform the job satisfactorily? If the district court was correct on that point also, we reach the third issue, the propriety of the relief.

The municipal defendants do not here challenge the findings of racially disproportionate impact, but the intervenors do. The basic facts are these: Roughly 11.5% Of the 14,168 applicants who entered the examination halls were black or Hispanic. Yet minority members comprised only 5.6% Of those who had passed the written, physical and medical examinations at the time of the hearing. Non-minority candidates thus survived the screening process at a rate more than twice that of minority candidates. Perhaps even more important, 18.4% Of the whites who took the examination ranked in the top 4000 and survived the physical^[5] while the comparable figure for minority candidates was 6.6%, a disparity of 2.8 to 1.

In challenging these statistics, the intervenors attack the reliability of the two ethnicity surveys by which the figures were gathered. The first was conducted by the Vulcan Society, an organization representing black firemen. The Society posted in front of the examination halls observers who counted the minority candidates as they entered. The Fire Department itself conducted the second survey, a 'sight survey' of those candidates who passed all stages of the selection process and were deemed 'finally qualified.' While the rather crude procedures of physical observation used in the surveys doubtless led to error in some cases, it is hard to believe that survey errors could have accounted for the striking racial imbalance that the results indicated. Indeed, it is arguable that the statistics probably underestimate the disparity between the percentage of minority candidates who took the examination and the percentage who qualified for appointment, since in the initial survey the Society instructed its observers to register candidates as minority members only when they were certain the candidates were black or Hispanic, but there is no indication that the Fire Department was similarly conservative in its survey of the candidates who qualified.

The intervenors also criticize the plaintiffs' comparison of the number of minority candidates who took the written examination with the number who finally qualified for appointment as a means of proving the racially disproportionate impact of the written test. The comparison was invalid, they claim, because minority members might have been disproportionately eliminated by either the physical or medical examinations, rather than by the written test. In theory this is true, and since there is no claim that the physical and medical examinations were biased, such a result would vitiate plaintiffs' constitutional claim. But plaintiffs' statistical expert produced an analysis showing that on the hypothesis that the written examination did not discriminate against minority applicants, the non-minority candidates must have passed the physical and medical examinations at a rate almost three times that of the minority candidates, a result he properly regarded as extremely unlikely. $yThe intervenors point to several other factors which they claim may undercut the validity of plaintiffs' statistics, but none of them casts serious doubt on the court's finding. They contend first that a substantial number of minority candidates may have done well on the written examination but been prevented from becoming 'finally qualified' by the diploma or height requirement or the criminal conviction bar. Yet since the factors requiring automatic disqualification were publicized prior to the administration of the examination, it seems unlikely that a substantial number of candidates took the written test in the face of certain rejection. Second, the intervenors point to the large number of candidates who passed the written test but did not appear for their medical or physical examinations; however, there is no reason to suppose that minority candidates were significantly overrepresented among the 'no-shows.' Finally, the intervenors claim that the 'character review' procedure may have been responsible for eliminating a large number of minority candidates. This claim is frivolous, as it appears that only four candidates had been eliminated by the character review screening process at the time of the hearing.

It may well be that the cited figures and other more peripheral data relied on by the district judge did not prove a racially disproportionate impact with complete mathematical certainty. But there is no requirement that they should. 'Certainty generally is illusion, and repose is not the destiny of man.'^[6] We must not forget the limited office of the finding that black and Hispanic candidates did significantly worse in the examination than others. That does not at all decide the case; it simply places on the defendants a burden of justification which they should not be unwilling to assume.

In Castro v. Beecher, supra, 459 F.2d at 732, the First Circuit stated that

The public employer must, we think, in order to justify the use of a means of selection shown to have a racially disproportionate impact, demonstrate that the means is in fact substantially related to job performance.

Judge Coffin later referred to the defendants' obligation to 'come forward with convincing facts establishing a fit between the qualification and the job.' Id. We do not consider that this court's references to 'a heavy burden' in Chance, supra, 458 F.2d at 1176, and Bridgeport Guardians, supra, 482 F.2d at 1337, meant anything more. The point we were endeavoring to underscore is that a showing of a racially disproportionate impact puts on the municipal or state defendants not simply a burden of going forward but a burden of persuasion. This may indeed prove to be 'heavy' because plaintiffs are likely to produce experts who will find numerous grounds for criticizing the examinations, whether legitimately or not. But if the public employer succeeds in convincing the court that the examination was 'substantially related to job performance,' an injunction should not issue simply because he has not proved this to the hilt.

We shall first consider the written examination. No one now challenges Judge Weinfeld's conclusion that Questions 81-100, entitled 'City Government and Current Events,' were not job related.^[7] Taking this as a given, appellants advance two major contentions: One is that the judge found nothing else wrong with the written examination, with the consequence that if he erred in insisting on a competitive physical, the only relief required would be to regrade the examination on the basis of the first 80 questions. The alternative contention is that if he found more to be wrong, he was in error. We reject both.

Although the judge placed particular emphasis on the unrelatedness of the civic affairs questions, this was not the limit of his criticism of the written examination. He sustained plaintiffs' contention that defendants failed to perform an adequate job analysis in preparing the examination^[8] and said that 'The record compels the conclusion that the procedures employed by defendants to construct Exam 0159 did not measure up to professionally accepted standards concerning content validity.' He added that 'Even if defendants were not required to conform precisely to all the requirements of a professional job analysis, it is clear that the methods actually employed were below those found unsatisfactory in Chance . . ..' Turning to defendants' contention that a faulty method of developing the examination should not be fatal if the result is satisfactory, he said that 'Even if this contention is accepted, under these circumstances only the most convincing testimony as to job-relatedness could succeed in discharging their (defendants') burden,' but that 'The testimony of defendants' expert . . . not only failed to meet this burden but even acknowledged the presence of a major flaw in the examination which is in itself fatal.' The 'major flaw' that the court pointed to was the inclusion of the twenty civics questions, but we do not think the mention of this serious defect indicated that, in the judge's view, the rest of the examination was without substantial flaws.

Cases like this one have led the courts deep into the jargon of psychological testing. Plaintiffs insist that the only satisfactory examinations are those which have been subjected to 'predictive validation' or 'concurrent validation,' preferably the former. The district court defined these terms as follows: 'Predictive validation consists of a comparison between the examination scores and the subsequent job performance of those applicants who are hired'; 'Concurrent validation requires the administration of the examination to a group of current employees and a comparison between their relative scores and relative performance on the job.' The judge wisely declined to insist on either. The Fourteenth Amendment no more enacted a particular theory of psychological testing than it did Mr. Herbert Spencer's Social Statics. Experience teaches that the preferred method of today may be the rejected one of tomorrow. What is required is simply that an examination must be 'shown to bear a demonstrable relationship to successful performance of the jobs for which it was used.' Griggs v. Duke Power Co., 401 U.S. 424, 431, 91 S.Ct. 849 at 853, 28 L.Ed.2d 158 (1971); McDonnell Douglas Corp. v. Green, 411 U.S. 792, 802 n.14, 93 S.Ct. 1817, 36 L.Ed.2d 668 (1973).^[9] To be sure, an impressive showing of predictive validation may end the inquiry then and there, and one of concurrent validation may come close to doing so; thus these methods may well be preferable in that sense. But these two schemes have their own difficulties,^[10] and the failure to use one of them is not fatal, at least from a constitutional standpoint, as long as the examination is properly job-related.

The intervenors argue that the judge confused 'construct validation' with 'content validation' and thereby erred in finding Exam 0159 insufficiently job-related. We do not think the judge was confused. He characterized 'content validation' in accordance with both prior case law and the leading authorities in the field:

An examination has content validity if the content of the examination matches the content of the job. For a test to be content valid, the aptitudes and skills required for successful examination performance must be those aptitudes and skills required for successful job performance.

See Bridgeport Guardians, supra, 482 F.2d at 1338. A simple example of an examination having content validity would be a typing test for applicants for a job as a typist. Although the court had no occasion to discuss 'construct validity,' that method is plainly distinguishable from 'content validity.' As noted in Bridgeport Guardians, supra, 482 F.2d at 1337-1338, 'construct validity' requires identification of general mental and psychological traits believed necessary to successful performance of the job in question. The qualifying examination must then be fashioned to test for the presence of these general traits. To design a 'construct valid' test for typists, the examiners would first determine that a typist's job requires, for instance, the ability to concentrate, perseverance, and attention to detail. Assuming that the identification of necessary traits was accurate, an examination that properly tested for those traits would have construct validity.

Instead of burying himself in a question-by-question analysis of Exam 0159 to determine if the test had construct or content validity, the judge noted that it was critical to each of the validation schemes that the examination be carefully prepared with a keen awareness of the need to design questions to test for particular traits or abilities that had been determined to be relevant to the job. As we read his opinion, the judge developed a sort of sliding scale for evaluating the examination, wherein the poorer the quality of the test preparation, the greater must be the showing that the examination was properly jobrelated, and vice versa. This was the point he made in saying that a showing of poor preparation of an examination entails the need of 'the most convincing testimony as to job-relatedness.' The judge's approach makes excellent sense to us. If an examination has been badly prepared, the chance that it will turn out to be job-related is small. Per contra, careful preparation gives ground for an inference, rebuttable to be sure, that success has been achieved. A principle of this sort is useful in lessening the burden of judicial examination-reading and the risk that a court will fall into error in umpiring a battle of experts who speak a language it does not fully understand. See Chance, supra, 458 F.2d at 1173.

The court's findings with respect to the construction of the examination were as follows:

The only witness who testified concerning the construction of Exam 0159 was Edward Scheinkman, not a Fire Department official, but Assistant Chief of the division of The Department of Personnel charged with responsibility for its preparation. Mr. Scheinkman began preparing the examination by gathering together the file on the previous examination, the former notice of examination, the class specifications (a very cursory description of the job contained in the notice of examination) and a magazine published by the Department. He contacted Chief Hartnett, who was then in charge of the Training Division of the Department, to inquire as to the Department's view of the areas of knowledge which should be included in the examination. Hartnett suggested that the subjects covered in the last test should be covered again, with the addition of a section on City government and current events. Somewhat significantly, Fire Commissioner Lowery, with his years of experience as a fire fighter and as an administrator, was not consulted as to the content of the written examination. Schinkman testified that he never performed any job analyses, did not know of any which were used in the preparation of Exam 0159 and that none were made while he was in the division which prepared examinations for the job of firemen.

Appellants do not seriously assert that these findings are clearly erroneous, and our examination of the record convinces us they are far from being so.

The judge was also warranted in rejecting the testimony of defendants' expert, Forbes McCann, that, except for the twenty civics questions, Scheinkman had achieved the miracle of stumbling into an examination that bore 'a demonstrable relationship to successful performance of the jobs' without having formulated an adequate analysis of just what the jobs were or what traits they demanded.

It is arguable that McCann's testimony proved the opposite of what he contended. Like Scheinkman, he insisted that the purpose of the test was to examine for the ability to learn to become a fireman in the probationary training school, not for the ability to perform the tasks required of a fireman. Performance on a written multiple-choice examination may well correlate quite highly with the ability to learn certain skills but not with the ability to perform them on the job.^[11] On the other hand, the defendants could respond that since the probationary training school is a necessary element in becoming a good fireman, the Department is justified in weeding out applicants who cannot benefit sufficiently from the training to be there afforded.

We prefer not to enter this morass, since there were ample grounds for rejecting McCann's testimony even if his premise were to be accepted. We cite a few: His assertion that ability to comprehend written materials was the most important single factor in a fireman's job is at war with common sense. His defense of the mathematics questions, despite his concession that very few firemen occupy jobs that require calculating skills, on the ground that there were only six or seven such questions on the test, ignored two important factors: One was that the scores on the examination were so closely bunched that a difference of only a few points could mean the difference of several thousand places on the eligibility list. The other was that since many other questions either were plainly invalid or too easy to have any differentiating effect,^[12] the six or seven mathematics questions actually constituted about 20% Of the resolving power of the test. The court was abundantly justified in accepting the criticism of this test by plaintiffs' experts and rejecting the defense.

The defendants and the intervenors ask us to set aside Judge Weinfeld's finding that Exam 0159 was insufficiently job-related because of the absence of a competitive, as distinguished from a merely qualifying, physical examination. We decline to do so.

We can speedily reject the first ground of attack, namely, the absence of evidence that the minority group candidates would do better than whites on a competitive physical examination. This misinterprets Judge Weinfeld's opinion. He did not hold that the use of a merely qualifying physical in itself necessarily or even probably worked against the minorities; what he held was that the absence of a competitive physical in the selection process for a largely physical vocation was additional evidence of the lack of job-relatedness of the selection procedure considered as a whole.

We likewise reject the claim that there was insufficient evidence to support this finding. Several witnesses testified to the high physical demands of a fireman's job. The Department had conducted competitive physical examinations from 1919 to 1968, and Fire Commissioner Lowery and Fire Chief O'Hagen expressed a strong and wellreasoned preference for the practice. While Captain Meyers supported the use of a qualifying physical, he conceded what indeed must be obvious even to judges, namely, that many of a fireman's duties are very strenuous and that the work requires substantial dexterity. The only truly contrary opinion was McCann's, and the court was warranted in considering his reasons to be unpersuasive. It is true that some of a fireman's duties, e.g., inspection, may require little or no physical prowess and that, as was urged by counsel for the intervenors, the intellectual content of a fireman's work may have increased far beyond that familiar in our youth. But that does not mean that no significant physical content remains.

We stress the limited nature of our holding. We do not read Judge Weinfeld as having said that if a written examination were sufficiently job-related, a competitive physical would always be constitutionally required, although he obviously would view such a physical with favor. There are considerations of cost and convenience that militate against giving a competitive physical to an extremely large group, including some who will rank so low on a proper written examination that even an Olympic score on a competitive physical would not put them within hiring range. Plaintiffs say these difficulties can be readily overcome, but they do not tell us how. In any event, there is no need to decide the question at this time. All that we regard the judge as having held, and all that we now approve, is that, incombination with the defects in preparation and content of Exam 0159 which we have described, the use of a merely qualifying physical examination rendered the Fire Department's selection procedures insufficiently job-related to withstand constitutional attack.

All parties object to the interim relief directed by the trial judge. What we have said sufficiently disposes of the objections of the municipal defendants and the intervenors based on the contention that the only defect in Exam 0159 was the 20 civics questions and that therefore the court should have done no more than to order that the examination be regraded. There remain the challenges to the court's use of an interim quota system and to the ratio of minority to non-minority candidates that the court selected. As in Bridgeport Guardians, supra, 482 F.2d at 1340, we approach the use of a quota system 'somewhat gingerly' and approve this course only because no other method was available for affording appropriate relief without impairing essential city services. As to the ratio chosen, the intervenors argue that the proper ratio should be 7 majority candidates to 1 minority candidate, and the plaintiffs contend that the ratio should be 1:1.^[13]

The argument for the 7:1 figure is that the ratio of whites to minority members who took the examination was 8:1 and that only a slight adjustment in that figure is needed to take account of improper disparities in appointments already made. Arguing against this and in support of a 1:1 ratio, plaintiffs urge that the nature of Exam 0159 and its predecessors had discouraged minority members from taking the examination; that although there was no specific evidence as to the discriminatory effect of previous examinations, on which Exam 0159 was patterned in considerable measure, this could be gleaned, sufficiently for the purpose of framing relief, from the fact that only 5% Of the New York City Fire Department consisted of minority members as against a 32% City-wide minority population in the eligible age group; and that at most the district court's 3:1 ratio up to June 30, 1974 would bring the percentage of minority firemen up to only 6.7%.

It is quite true that the judge's 3:1 ratio does not purport to rest on any scientific basis. But neither does the plaintiffs' proposal, and the intervenors' figure does not take account of factors the judge was permitted to consider. In arriving at a ratio midway between what would have been appropriate on the basis of correcting the inequities of Exam 0159 alone and the plaintiffs' demands for much more extensive relief, the judge took appropriate account both of the resentment of non-minority individuals against quotas of any sort and of the need of getting started to redress past wrongs. See Louisiana v. United States, 380 U.S. 145, 154, 85 S.Ct. 817, 13 L.Ed.2d 709 (1965); United States v. Wood, Wire & Metal Lathers, Local 46, 471 F.2d 408, 413 (2 Cir.), cert. denied, 412 U.S. 939, 93 S.Ct. 2773, 37 L.Ed.2d 398 (1973). As the Supreme Court has stated, 'The framing of decrees should take place in the District rather than in Appellate Courts.' International Salt Co. v. United States, 332 U.S. 392, 400, 68 S.Ct. 12, 17, 92 L.Ed. 20 (1947); Chance, supra, 458 F.2d at 1178. A case like this, involving interim relief expected to be required for less than a year and reflecting the wisdom of a widely experienced and highly respected judge, should be almost the last to attract appellate intervention.

We turn now to the portion of plaintiffs' cross-appeal concerning the judge's failure to act on their complaints concerning the requirements for a minimum height and a high school or high school equivalency diploma, and the bar against applicants convicted of a felony or of petty larceny. Plaintiffs urge that all three automatic disqualifications operate more harshly on blacks and/or Hispanics than on whites and therefore require a showing of job-relatedness.

The refusal to rule on the minimum height requirement, despite the fact that one of the plaintiffs would have been barred thereby, seems to have been the result of a misunderstanding. The preliminary injunction sought by the plaintiffs was addressed to the eligibility list prepared as a result of Exam 0159, and the testimony had been confined to the racial effect and the job-relatedness of the written and physical examinations. It was altogether appropriate for the judge to seek agreement of the parties that he might rule on these issues as on final hearings and, having obtained this, to make a disposition of them. But, in the absence of an agreement by plaintiffs' counsel to abandon the minimum height issue, which was squarely raised as to one of the plaintiffs, the judge was obliged to rule on the question either one way or the other.^[14] Plaintiffs' counsel denies having given such consent, and we find none. Accordingly, we remand for disposition, with a hearing if necessary, on this portion of the complaint.

Plaintiffs' position on the diploma requirement and conviction bar is not so strong since none of them were affected by these provisions. But, here again, the judge's commendable desire to get at the heart of the complaint seems to have created a bit of a procedural impasse. He was entirely right in thinking it unnecessary, from the plaintiffs' standpoint, for him to decide on class action designation in order to pass upon the issues raised in regard to Exam 0159.^[15] If the examination procedures were found unconstitutional as regards the named plaintiffs, they were equally so as regards all eligible blacks and Hispanics, and it would be unthinkable that the municipal defendants would insist on other actions being brought. But the correct decision of that point did not relieve the judge from deciding whether plaintiffs should be allowed to represent all blacks and Hispanics who might qualify as firemen but for the requirements here considered, even though the individual plaintiffs were not personally affected by them.

We do not find that the cases cited by plaintiffs, either under 1983 or under Title VII of the Civil Rights Act, e.g., Johnson v. Georgia Highway Express, Inc., 417 F.2d 1122 (5 Cir. 1969); Parham v. Southwestern Bell Tel. Co., 433 F.2d 421 (8 Cir. 1970); Carr v. Conoco Plastics, Inc., 423 F.2d 57 (5 Cir.), cert. denied, 400 U.S. 951, 91 S.Ct. 241, 27 L.Ed.2d 257 (1970); and Castro v. Beecher, supra, mandate an answer of that question in their favor.^[16] Indeed, Castro v. Beecher, supra, 459 F.2d at 732 & n.8, seems to point the other way. We therefore prefer not to decide the issue but rather to remand this also to the district court. If the court determines not to accord class action designation, it should consider whether plaintiffs should not be allowed to amend their complaint to join additional persons who would be affected by the diploma requirement and the conviction bar. If we may be pardoned for speaking practically, we cannot understand why the municipal defendants should resist such an amendment. Much work has already been done on these points. It is evident that they will be raised sooner or later, and the City's interests would seem to be served by having them determined sooner rather than run the risk that a new and otherwise unobjectionable selection procedure might be invalidated on one of these grounds. In saying this we are in no way intimating a view that the diploma requirement or the conviction bar is invalid; we say only that it is in everyone's interest that questions about them should be promptly resolved.

The judgment is affirmed except that the cause is remanded for further proceedings on the issues discussed in Part VI of this opinion. Plaintiffs may recover costs against the municipal defendants.

NOTE: OPINION CONTAINS TABLE OR OTHER DATA THAT IS NOT VIEWABLE