Only two white soldiers were executed for rape or attempt on colored women. Not to say that only white women were raped during the CW but if complaint from white women were rare, complaint from blacks were extremely rare...But rape of black women , mostly slaves (not protected by laws against sexual agressions in the antebellum South) was not unknown and probably more common that on white women according to the historians interested by this aspect of  the war.  It's possible to find some interesting Union reports in the Official records about this subject :

• "I found the prejudice of color and race here in full force, and the general feeling of the army of occupation was unfriendly to the blacks. It was manifested in various forms of personal insult and abuse, in depredations on their plantations, stealing and destroying their crops and domestic animals, and robbing them of their money.The women were held as the legitimate prey of lust, and as they had been taught it was a crime to resist a white man they had not learned to dare to defend their chastity". (Brigadier Gl Saxton, USA, Beaufort, SC, 30 december 1864 to E. Stanton, OR SIII volV p1028/29)
  

• "GENERAL: I feel surprised, after the precautions that have been taken by yourself and officers, to find that many depredations have been committed near this place, and certain things done that would disgrace us even in the enemy's country, e.g., the robbing of some negroes and abusing their women." (Gl Howard, 10 january 1865, beaufort, SC, OR SI, vol XLVII pt 2 p33).
  

• "Reports are made to me of the most disgraceful outrages on the part of troops along the road within 10 or 12 miles of your station. Not only is property taken without vouchers, as required by law and my repeated orders, but property is wantonly destroyed, negro women are debauched, and ladies insulted. Such acts are said to have been committed at Mr. Clay's place, 10 miles west of you, yesterday or the day before."  Gl Buell from Huntsville, 12 august 1862. (OR SI volXVI pt 2 p319)
  

• "When there is added to this the irregularities of the soldiery--such as taking poultry, pigs, milk, butter, preserves, potatoes, horses, and in fact everything they want; entering and searching houses, and stealing in many cases; committing rapes on the negroes and such like things--the effect has been to make a great many Union men inveterate enemies, and if these things continue much longer, our cause is ruined." in a letter from John T.K Hayward, unionist agent in Missouri to Simon Cameron, 13 august 1861 (OR SI, volIII p459)
  

• In North Carolina, a black man named John Allen, beaten and bleeding from the head, rushed into camp begging for help. An officer heard the row and came out of his tent to investigate the matter . He heard Allen explain that a soldier had beaten him and his wife and was attempting to rape his wife. A short distance from Allen's house soldiers arrested an Illinois corporal A.C.Warner. During the trial Allen was introduced like a witness but the judge advocat objected that " this witness had not sufficient intelligence to comprehend the nature of the oath" , jugdes concurred without questionning Allen and the court dismissed all charges against Warner !  RG 153 NA MM3937