The incident happened just after 11 a.m. in the 600 block of John Street, and was visible from the KOMO News offices at nearby Fisher Plaza.
Seattle Fire Department spokesman Kyle Moore said the man fell 50 feet from some scaffolding on one of the upper floors of the building.
Workers and medics immediately began CPR on the 35-year-old man, who was unconscious after the fall. Medics continued CPR as he was rushed by ambulance to Harborview Medical Center.
Doctors were unable to save the man and he died at the hospital.
Co-workers say the man was a welder and just getting his bearings at the job site.
"I guess he'd been with the company, with SRM, I guess for about two months, but today was his first day here," said David Widmer.
Safety protocols require such workers to be clipped into a harness, but witnesses say they didn't see anything attached to the man when he fell.
Moore said the accident likely will be investigated by the state Department of Labor and Industries and possibly by the U.S. Occupational Safety and Health Administration.
State inspectors checked the job site in April and found no safety violations. The company running the project also has a good record, with no license violations in the past six years.