Lumber
Lumber is one of the forms of building materials and has been part of the games since Harvest Moon on the SNES.
Lumber is accumulated by chopping down trees, e.g. a whole tree will get you around 8+ pieces.
Material Stone
In some of the more recent games, Stone has been available for building with. For example, in Harvest Moon: Tree of Tranquility, material stone can be accumulated by smashing boulders. Each boulder gives about 4 pieces.