The Kelly House is set on a flat site with a lawn at the rear. The house consists of a two-story, gable-roofed structure with a one-story, gable-roofed ell that originally contained the garage. The house has been heavily altered with poorly designed additions. The main part of the house is clad in wood clapboards, while the ell and altered areas are clad in V-channel vertical wood siding. A freestanding, gable-roofed, two-car garage with an open shed at the rear is set next to the house. Unlike most of the Modern houses in New Canaan, the Kelly House was designed as a traditional Colonial Revival structure with some vaguely Modern elements, most notably in the design of the fenestration. The hopper windows on the first floor have heavy, strongly defined mullions and are either grouped into squared-off bay windows or are adjacent to fixed sash or doors.
The Kelly House was designed by architect Bimel Kehm as a speculative house. In 1954, Bimel Kehm and R.R. Austin acquired the property from West Hills of New Canaan, which may have been a development company; a note on the assessor records states "West Hills Dev." and the street appears to be a post-World War II development. Kehm designed a one-family house with incorporated garage for the site. By October 1954, the house was 40% finished and was completed by 1955.
The house was sold to Dean McCune Kelly in 1955. Between 1958 and 1959, the incorporated garage was converted into a playroom (the original door openings are now filled with large windows) and a new freestanding two-car garage with attached open shed was completed. In 1961, the house was purchased by Vincent A. and Ruth M.W. Tauber. In 1968, the screened porch at the rear was enclosed and converted into living space. At least some of the original porch columns are still visible on the interior. The fieldstone patio at the back may have also been installed at this time, since it does not appear in earlier assessor records. In 1987, a second floor was added above the former screened porch. Around 1992, a small 12'x8' addition was built at the side of the house; it appears that the window on this addition was moved from another location on the house since it matches the original windows and the moldings do not line up with the window moldings on the adjacent fa?ade. Kevin M. and Sally S. Sweeney purchased the house in 1992.