The house where the Lebensborn Party takes place is clearly north-American, the distinctive architecture of the house with a horizontal stacked board construction and the external chimney is something that cannot be found around Berlin.
Tagomi interacts with people of the "real" 1962 time line without changing attire, which confuses his son. However, nobody seems to notice the very distinctive government lapel pin of the Japanese Pacific States. In the heavily politically charged environment of the Cold War at the height of the Cuban Missile Crisis, an unfamiliar insignia on a foreign man would, at the very least, raise some questions, if not downright arrest and interrogation.