Macy's was founded by Rowland Hussey Macy, who between 1843 and 1855 opened four retail dry goods stores, including the original Macy's store in downtown Haverhill, Massachusetts, established in 1851 to serve the mill industry employees of the area. They all failed, but he learned from his mistakes. Macy moved to New York City in 1858 and established a new store named "R.H Macy Dry Goods" on Sixth Avenue between 13th and 14th Streets, which was far north of where other dry goods stores were at the time. On the company's first day of business on the 28th of October 1858 sales totaled US$ 11.08, equal to $279.82 today. From the very beginning, Macy's logo has included a star in one form or another, which comes from a tattoo that Macy got as a teenager when he worked on a Nantucket whaling ship.

As the business grew, Macy's expanded into neighboring buildings, opening more and more departments and used publicity devices such as a store Santa Claus, themed exhibits and illuminated window displays to draw in customers. It also offered a money back guarantee, although it only accepted cash into the 1950's. The store also produced its own made-to-measure clothing for both men and women, assembled in an on-site factory. The store later moved to 18th Street and Broadway, on the "Ladies' Mile", the elite shopping district of the time, where it remained for nearly forty years.

In 1875, Macy took on two partners, Robert M. Valentine (1850-1879), a nephew; and Abiel T. La Forge (1842-1878) of Wisconsin, who was the husband of a cousin. Macy died just two years later in 1877 from Bright's disease. The following year La Forge died, and Valentine died in 1879. Ownership of the company was passed down through the Macy family until 1895, when the company, now called "R. H. Macy & Co.", was acquired by Isidor Straus and his brother Nathan Straus, who had previously held a license to sell china and other goods in the Macy's store.

In 1902, the flagship store moved uptown to Herald Square at 34th Street and Broadway, so far north of the other main dry goods emporia that it had to offer a steam wagonette to transport customers from 14th Street to 34th Street. Although the Herald Square store initially consisted of just one building, it expanded through new construction, eventually occupying almost the entire block bounded by Seventh Avenue on the west, Broadway on the east, 34th Street on the south and 35th Street on the north, with the exception of a small pre-existing building on the corner of 35th Street and Seventh Avenue and another on the corner of 34th Street and Broadway. This latter 5-story building was purchased by Robert H. Smith in 1900 for $375,000 - an incredible sum at the time - with the idea of getting in the way of Macy's becoming the largest store in the world: it is largely supposed that Smith, who was a neighbor of the Macy's store on 14th Street, was acting on behalf of Siegel-Cooper, which had built what they thought was the world's largest store on Sixth Avenue in 1896. Macy's ignored the tactic and simply built around the building, which now carries Macy's "shopping bag" sign by lease arrangement.

The original Broadway store, designed by architects De Lemos & Cordes and built in 1901-1902 by the Fuller Company. It has a Palladian facade, but has been updated in many details. Other additions to the west were added in 1924 and 1928, and the Seventh Avenue building in 1931, all designed by architect Robert D. Kohn, the newer buildings becoming increasingly Art Deco in style. The building was added to the National Register of Historic Places as a National Historic Landmark in 1978. The store has several wooden escalators still in operation.